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EXECUTIVE  SUMMARY 


The  Air  Force  Directorate  of  Maintenance  (USAF/ILM)  and  the  Standard 
Systems  Group,  Maintenance  Systems  Division  (HQ  SSG/ILM)  sponsored  and 
coordinated  several  implementation  and  evaluation  efforts  for  improving  performance  at 
the  Point-of-Maintenance  (POMx).  The  Air  Force  Automatic  Identification  Technology 
Program  Management  Office  (AIT  PMO)  was  responsible  for  implementing  an  AIT 
system  for  improving  flightline  data  collection  activities  at  Hurlburt  Field  AFB,  FL. 
Evaluations  included  testing  of  voice  recognition  on  the  flightline  for  work  order 
management  and  aircraft  parts  ordering.  As  part  of  the  POMx  efforts,  the  Air  Force 
Research  Laboratory,  Logistics  Readiness  Branch  (AFRL/HESR)  funded  a  series  of 
Human  Factors  usability  tests  for  the  purpose  of  evaluating  the  maintenance  hardware 
devices  to  be  used  at  the  Point-of-Maintenance.  Three  usability  tests  were  conducted: 
Spiral  1  Usability  Test,  Spiral  3  Synthetic  Usability  Test,  and  Spiral  3  Field  Usability 
Test  (Note,  Spiral  2  did  not  include  usability  testing  and  results  are  not  included  in  this 
report).  The  purpose  of  the  Spiral  1  testing  was  to  evaluate  various  hardware  platforms 
for  their  potential  usability  on  the  flightline.  The  overall  purpose  of  the  Spiral  3  testing 
(both  the  synthetic  and  field  tests)  was  to  evaluate  the  use  of  potential  POMx  target 
platforms  for  presenting  technical  data  on  the  flightline  (see  Figure  1).  All  testing  was 
conducted  at  the  16th  SOW,  Hurlburt  Field  AFB,  FL. 


Spiral  Process 


•  Flightline  Field  Test 

•  Portable  and 
Wearable 
Computers 

•  Opening  Work 
Orders 

•  Hardware  Usability 


•  In-Unit  Synthetic 
Test 

•  Three  Screen  Sizes 

•  10  Types  of 
Technical  Manuals 

•  Tech  Data  Usability 
on  Hardware 


•  Flightline  Field  Test 

•  Best  Devices  for 
viewing  Technical 
Manuals 

•  Perform 
Maintenance  Task 

•  Tech  Data  Usability 

with  Hardware  on 
the  Flightline _ 


Spiral  1 


Spiral  3,  Synthetic  Test 


Spiral  3,  Field  Test 


Figure  1.  Spiral  Development  Process 

The  Spiral  1  Usability  Test  compared  5  computing  systems  for  usability  in  an  Air 
Force  flightline  environment.  The  purpose  of  the  evaluation  was  to  compare  the 
platforms  for  opening  work  orders  from  the  aircraft  location.  The  five  devices  consisted 
of:  1)  a  Motorola  handheld  radio  (on  which  work  orders  were  opened  via  voice 
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commands),  2)  an  Itronix  ruggedized  computer,  3)  a  Libretto  miniature  computer,  4)  a 
Xybemaut  wearable  computer  with  a  head-mounted  display,  and  5)  a  Xybemaut  with  a 
wrist-worn  display.  Crew  Chiefs  and  Communication/Navigation  Specialists  identified 
that,  among  the  devices  shown,  the  Libretto  computer  was:  1)  rated  highest  for  overall 
ease-of-use,  and  2)  the  preferred  technology  overall.  This  device  is  small  and 
lightweight;  it  offers  a  full  keyboard  and  full  screen  that  technicians  can  carry  in  one 
hand.  The  Spiral  1  study  also  revealed  additional  capabilities  desired  by  technicians,  not 
provided  by  the  Libretto,  these  included  voice  recognition  and  touch  screen. 

The  Spiral  3  Synthetic  Usability  Test  compared  3  computing  systems  for  their 
usability  for  viewing  technical  manuals.  In  order  to  control  environmental  variables  such 
as  lighting  and  glare  on  ability  to  view  technical  manuals,  a  synthetic  test  was  conducted 
inside  the  maintenance  unit  building.  Testing  occurred  in  the  unit  conference  room  and 
attempted  to  simulate  flightline  maintenance  tasks  in  this  synthetic  setting.  The  three 
hardware  devices  included  a  full-screen  Panasonic  Toughbook,  a  half-screen  LXE-MX3, 
and  a  quarter  screen  Intermec  710.  The  LXE-MX3  and  the  Intermec  710  were  selected 
for  this  test  due  to  their  inclusion  in  the  POMx  implementation  effort  headed  up  by  the 
Air  Force  AIT  PMO.  The  Panasonic  Toughbook  was  selected  due  to  its  advanced 
capability  as  a  ruggedized  computing  system  and  based  on  findings  from  Spiral  1  testing. 
Ten  different  samples  of  technical  data  layouts  were  presented  to  Crew  Chiefs  and 
Specialists  for  evaluation.  Users  were  required  to  evaluate  the  10  types  of  technical  data 
layouts  for  various  usability  issues.  Technical  data  samples  included:  work  cards,  job 
guides,  fault  report  manuals,  fault  isolation  trees,  and  schematics.  While  evaluation  of  all 
these  types  of  formats  was  conducted,  analysis  of  the  results  focused  solely  on  ability  to 
view  Job  Guides.  Among  the  devices  shown,  the  Panasonic  Toughbook  was  rated  as 
acceptable  for  viewing  Job  Guides  and  ratings  were  significantly  better  for  this  device 
than  for  the  other  two.  While  some  usability  issues  were  identified  for  the  LXE-MX3 
and  the  Intermec  710,  all  three  devices  were  selected  for  the  Spiral  3  Field  Usability  Test. 

The  Spiral  3  Field  Usability  Test  compared  3  computing  systems  for  their 
usability  for  viewing  Job  Guides  on  the  flightline.  Flightline  testing  included  performing 
the  Power-On  maintenance  task.  Hardware  devices  included  a  full-screen  Panasonic 
Toughbook,  a  half-screen  LXE-MX3,  and  a  quarter  screen  Intermec  710.  Technicians 
included  Crew  Chiefs  and  Specialists.  For  flightline  viewing  of  the  Power-On 
maintenance  task,  technicians  preferred  the  Intermec  710  quarter  screen  device.  The 
quarter  screen  display  on  this  small,  lightweight  device  allowed  technicians  to  complete 
the  Power-On  task.  Note  that  while  the  Intermec  was  the  most  preferred  device  overall, 
several  qualifications  are  indicated  for  the  usability  of  the  device.  There  are  potential 
usability  problems  using  the  device  in  sunlight  conditions,  and  the  display  quality  was  not 
as  good  as  the  Toughbook.  Finally,  the  test  did  not  evaluate  performance  accuracy  (e.g., 
did  the  technician  miss  steps,  or  perform  steps  incorrectly)  when  using  this  device.  This 
type  of  accuracy  test  should  be  conducted  prior  to  implementing  Job  Guides  on  the 
flightline  using  a  !4  screen  device,  such  as  the  Intermec. 

In  conclusion,  usability  tests  showed  that  technicians  prefer  small,  lightweight 
devices,  such  as  the  Libretto  or  Intermec  710  for  several  types  of  tasks  performed  on  the 
flightline.  Features  such  as  touch  screen,  voice  recognition,  and  screen  quality  are  major 
considerations  for  selection  of  hardware  platforms  to  be  used  at  the  point  of  maintenance. 
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INTRODUCTION 


The  Point-of-Maintenance  (POMx)  usability  tests  consisted  of  human  factors  and 
user-centered  evaluations  of  the  POMx  activity.  These  experiments  emphasized 
maintainer  feedback  on  the  usability  of  POMx  applications  in  U.S.  Air  Force  aircraft 
maintenance  environments.  In  general,  tests  frequently  focus  on  maintainer  performance 
with  evolving  technologies  to  determine  whether  efficiencies  can  be  achieved;  however, 
usability  of  these  technologies  is  also  critical  in  assuring  efficient  operations.  If  the 
system  is  not  usable,  performance  enhancements  cannot  be  achieved.  The  tests 
conducted  in  the  current  evaluation  were  focused  on  the  usability  of  evolving  flightline 
maintenance  technologies. 

User  level  evaluations  were  conducted  to  determine  the  usability  of  POMx 
devices  in  presenting,  modifying,  and  transmitting  maintenance  and  service  data.  These 
evaluations  were  administered  to  technicians  in  real-world  and  synthetic  maintenance 
environments.  The  purpose  of  the  evaluations  was  to  compare  hardware  platforms  for 
use  on  the  flightline.  Usability  test  methods  used  for  the  current  evaluation  are  based  on 
principles  outlined  by  Dumas  and  Redish  (1993).  In  accordance  with  usability  testing 
methods  outlined  by  Dumas  &  Redish,  design  of  the  current  usability  evaluation  applied 
a  three-phased  approach  (see  Figure  2). 


Concerns 


V 


Specific 

Concerns 


M 


Metric: 


Figure  2.  Usability  testing  methods  (Dumas  and  Redish,  1993) 

For  each  evaluation,  the  three-phased  method  started  with  identification  of 
general  goals  or  areas  of  concern.  These  general  areas  of  concern  were  initially  specified 
by  the  proposal  briefings  for  POMx.  Air  Force  Research  Laboratory  (AFRL)  POMx 
participants  then  further  refined  each  of  the  general  concerns  into  a  set  of  specific 
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concerns  that  could  be  tested.  These  specific  concerns  provided  the  foundation  for 
developing  metrics.  Each  metric  was  designed  to  test  a  specific  usability  issue. 

Usability  Test  Objectives 

Three  usability  tests  were  conducted:  Spiral  1  Usability  Test,  Spiral  3  Synthetic 
Usability  Test,  and  Spiral  3  Field  Usability  Test.  The  purpose  of  the  first  spiral  of  testing 
was  to  evaluate  various  hardware  platforms  for  their  potential  usability  on  the  flightline. 
The  overall  purpose  of  the  third  spiral  (both  the  synthetic  and  field  tests)  was  to  evaluate 
the  use  of  potential  POMx  platforms  for  presenting  technical  data  on  the  flightline  (see 
Figure  1  in  Executive  Summary).  All  testing  was  conducted  at  the  16th  SOW,  Hurlburt 
Field  AFB,  FL.  General  areas  of  concern  for  each  test  are  listed  below. 

Spiral  1.  Usability  Test 

•  View  Aircraft  Status 

•  Create/Open  Work  Order  via  Voice  Recognition  Tool 

•  Create  Work  Order  using  PM  A  via  a  keyboard 

•  Determine  user  preferences  and  concerns  for  portable  computers  on  the 
flightline 

Spiral  3.  Synthetic  Usability  Test 

•  Comparative  ease  of  use  of  varying  screen  sizes  of  portable  computers 

•  Comparative  ease  of  use  of  tech  data  page  layout 

Spiral  3.  Field  Usability  Test 

•  Comparison  of  display  requirements  for  each  device 

•  Comparison  of  data  manipulation  devices  for  each  device 

•  Comparison  of  physical  characteristics  of  each  device 

Usability  Test  Hardware 

Hardware  was  carefully  selected  for  each  test.  The  Spiral  1  hardware  selection 
process  started  with  an  evaluation  of  various  mobile  and  wearable  hardware  platforms 
available  at  the  time  of  the  test.  The  chosen  hardware  represented  an  array  of  mobile  and 
wearable  computer  platforms  (See  Appendix  A  for  specification  of  all  hardware  used  in 
these  tests).  Due  to  computer  memory  and  infrastructure  limitations,  the  array  was 
limited  to  full  computers  running  Windows  2000  (i.e.,  no  handheld  computers  were 
used),  and  a  brick  radio.  The  computers  included  a  ruggedized  full-sized  notebook 
(frequently  identified  by  the  maintenance  community  as  the  type  of  device  needed  on  the 
flightline),  a  miniature  notebook  (to  determine  if  a  smaller  size  was  a  desirable 
characteristic),  a  wearable  computer  with  wrist-mounted  display  (allowing  hands  free 
movement  without  restricting  head  movement),  a  wearable  computer  with  a  head- 
mounted  display  (allowing  hands  free  movement  and  information  always  in  field  of 
vision).  This  array  of  hardware  allowed  for  evaluations  to  include  characteristics 
common  to  many  types  of  mobile  and  wearable  computer  systems. 

Hardware  selected  for  the  Spiral  3  tests  (both  synthetic  and  field)  were  selected 
based  on  two  primary  criteria:  1)  the  hardware  needed  to  include  items  being 
implemented  by  the  AIT  PMO  and,  2)  hardware  needed  to  include  desirable 
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characteristics  identified  in  the  Spiral  1  test.  The  LXE-MX3  and  the  Intermec  710  were 
two  devices  being  implemented  by  the  AIT  PMO  that  also  met  some  of  the  needs 
identified  in  Spiral  1.  For  example,  they  both  were  small,  lightweight  devices  and  had 
touch  screen  capabilities.  The  Panasonic  Toughbook  provided  excellent  screen  quality 
and  touch  screen  capabilities  in  a  full  computer. 

Usability7  Test  Design 

The  Air  Force  Research  Laboratory  Logistics  Readiness  Branch  (AFRL/HESR) 
has  developed  a  phased  approach  to  testing,  called  the  LSF  approach:  laboratory  testing 
(L),  synthetic  environment  testing  (S),  and  field-testing  (F)  (Quill,  Kancler,  Revels  & 
Masquelier,  1999).  These  three  phases  build  on  results  found  in  the  previous  phase  to 
capitalize  on  the  strengths  of  empirical  and  usability  testing  techniques.  For  example, 
results  from  the  laboratory  test  help  structure  the  design  of  the  synthetic  environment  test, 
etc. 

The  purpose  of  the  laboratory  evaluation  is  to  collect  empirical  data  on  individual 
parts  of  the  task.  For  example,  laboratory  tests  are  conducted  on  new  hardware  and 
software  configurations.  A  laboratory  test  may  compare  two  types  of  control  input 
devices  to  see  which  is  most  compatible  for  interacting  with  maintenance  technical  data. 
No  laboratory  tests  were  conducted  as  part  of  the  current  effort. 

The  synthetic  environment  test  provides  empirical  data  on  the  interaction  of  a  few 
key  elements  in  a  controlled  setting.  Selection  of  the  appropriate  elements  to  test  and 
appropriate  representation  of  those  elements  is  the  most  important  step  in  identifying  how 
the  parts  will  interact  in  the  real-world  environment.  This  test  requires  enough  subjects 
to  perform  statistical  analyses  required  by  the  empirical  test  (e.g.,  analysis  of  variance). 
The  Spiral  3,  Synthetic  Usability  Test  served  as  the  synthetic  environment  test  for  the 
current  effort. 

The  purpose  of  a  field  test,  as  defined  by  the  LSF  approach,  is  to  supplement 
information  found  in  the  synthetic  environment  by  providing  real-world  feedback  about 
the  task.  In  the  field  test,  all  elements  are  synthesized  to  recreate  the  system  in  its  actual 
working  environment.  While  collecting  objective  data  is  possible  in  a  field  test,  it  is 
often  very  difficult  to  control  extraneous  factors  in  the  actual  environment.  Therefore, 
the  field  test  is  used  primarily  to  collect  subjective  feedback  on  the  system.  Fewer 
subjects  are  generally  used  for  this  type  of  usability  test.  Virzi  (1992)  identifies  that  4  to 
5  subjects  identify  80%  of  the  usability  problems  with  a  system,  and  that  additional 
subjects  are  less  and  less  likely  to  identify  any  new  problems.  Usability  inspection 
methods  are  the  focus  of  these  tests,  not  empirical  data  collection  methods.  Field  tests 
conducted  in  the  POMx  usability  test,  included  the  Spiral  1  Usability  Test  and  the  Spiral 
3,  Field  Test. 


Background 

Since  1990,  the  Air  Force  Research  Laboratory,  Logistics  Readiness  Branch 
(AFRL/HESR)  has  been  researching  portable  and  wearable  computing  systems  and  their 
peripheral  devices.  Research  methods  including  demonstrations,  empirical  tests,  and 
usability  inspection  methods  have  provided  abundant  data  pertaining  to  the  usability  and 
usefulness  of  computing  systems  for  aircraft  maintenance.  Karat  (1994)  notes  that  as  a 
system  evolves,  it  is  necessary  that  research,  design,  and  development  efforts  include  all 
three  of  these  types  of  research  methods  (demonstrations,  empirical  tests,  and  usability 
inspection  methods).  The  following  paragraphs  review  empirical  studies  conducted  at 
AFRL/HESR,  and  emphasize  the  need  for  usability  testing  of  these  devices  as  well. 

Over  the  past  decade,  AFRL  has  demonstrated  portable  and  wearable  computer 
technologies  in  numerous  settings.  The  purpose  of  these  demonstrations  was  to  allow 
potential  users  to  perform  an  informal  walkthrough  using  the  mobile  computing  system. 
Potential  users  provided  informal  feedback  pertaining  to  the  potential  usability  of  the 
hardware  and  software  components.  In  addition  to  these  demonstrations,  many  field  tests 
have  been  conducted  which  have  provided  both  empirical  and  usability  inspection  data. 
Results  of  selected  studies  are  summarized  below. 

Empirical  studies  conducted  by  the  AFRL/HESR  have  evaluated  both  portable 
maintenance  aids  and  wearable  computers  for  flightline  maintenance.  Both  platforms 
have  been  shown  to  improve  maintenance  performance  over  use  of  paper-based  platforms 
(Carlson,  Smith,  Smith,  Thomas,  &  Smillie,  1992;  Thomas,  1995;  Friend  &  Grinstead, 
1992).  In  addition,  empirical  testing  of  performance  has  shown  that  wearable  computing 
systems  result  in  better  performance  than  portable  maintenance  aids  under  certain 
maintenance  conditions  (Friend  &  Grinstead,  1992).  These  studies  have  shown  that 
technicians  perform  more  quickly  and  accurately  when  using  a  wearable  computer  than 
when  using  a  portable  maintenance  aid  in  certain  settings.  While  these  empirical  studies 
have  compared  wearable  computer  technologies  with  portable  computers,  usability 
inspection  methods  (such  as  usability  testing)  have  not  compared  these  devices.  In  other 
words,  while  the  technician’s  performance  may  be  improved  it  has  not  been  determined 
how  usable  wearable  computers  are  in  comparison  with  portable  aids.  The  purpose  of  the 
current  study  was  to  make  these  types  of  usability  comparisons. 

In  1991,  a  study  was  conducted  that  compared  two  hardware  visual  displays 
(Masquelier,  1991).  The  study  compared  use  of  a  light  emitting  diode  (LED),  monocular, 
head  mounted  display  (the  Private  Eye™)  with  use  of  a  standard  Grid™  laptop  computer 
monitor  (set  on  a  table  top).  In  addition  to  this  hardware  comparison,  Masquelier 
evaluated  the  interaction  of  the  hardware  with  maintainers.  Two  groups  of  maintenance 
technicians  (experienced  and  inexperienced)  participated  in  the  study — sixteen  subjects 
in  total.  Each  subject  performed  two  avionics  communications  tasks  in  an  intermediate 
level  maintenance  task  environment.  Subjects  sat  at  a  workbench  while  performing 
inspection  and  fault  isolation  maintenance  on  computer  circuit  boards — mobility  around 
the  flightline  was  not  an  issue  in  this  test.  Software  was  standardized  for  both  displays 
and  the  presentation  interface  was  consistent  with  paper  technical  manual  presentations. 
Significant  differences  between  the  two  display  types,  maintainers,  and  interactions  were 
not  found  for  measures  of  either  task  performance  times  or  number  of  errors  made. 
Recommendations  for  follow-on  experiments  stated  that  similar  tasks  in  environments 
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requiring  movement  in  and  around  the  aircraft  might  be  more  appropriate  for  the 
wearable  computer  display. 

Acting  on  Masquelier’s  recommendation,  in  1992  Friend  and  Grinstead  conducted 
a  study  that  compared  two  similar  types  of  display  devices  in  a  flightline  maintenance 
environment.  In  the  flightline  task,  technicians  were  required  to  move  around  the 
airplane  conducting  fault  isolation  and  inspection  tasks.  To  provide  mobility,  a  portable 
computer  was  used  for  the  “standard”  display  device  while  a  wearable  computer  was 
used  with  the  head-mounted  display  (HMD).  Both  displays  were  liquid  crystal  displays. 
The  HMD  was  built  by  Intervision™.  In  keeping  with  Masquelier’s  design,  two  levels  of 
experience  were  selected — experienced  and  inexperienced. 

In  contrast  with  Masquelier’s  study,  the  results  of  this  study  yielded  several 
significant  effects  and  interactions.  For  the  fault  isolation  task,  inexperienced  technicians 
took  longer  to  isolate  the  defect  when  using  the  portable  computer;  whereas,  experienced 
technicians  performed  equally  well  using  either  display  device  (portable  or  HMD). 
Additionally,  in  the  inspection  task,  experienced  technicians  found  more  faults  (i.e., 
fewer  diagnostic  errors)  when  using  the  HMD  than  when  using  the  portable  computer; 
whereas,  inexperienced  technicians  performed  equally  with  either  device.  The  main 
effect  of  display  type  showed  technicians  using  the  HMD  found  five  more  faults,  on 
average,  than  their  portable-equipped  counterparts. 

The  overall  results  showed  that  when  comparing  a  wearable  computer  with  a 
portable  one,  performance  was  better  in  some  circumstances  with  the  wearable  computer. 

In  addition  to  these  empirically-based  tests,  AFRL/HESR  has  conducted  several 
usability  studies.  One  such  usability  study  evaluated  a  monocular,  occluding  head- 
mounted  display.  The  usability  study  included  evaluation  of  the  HMD  in  a  variety  of 
flightline  conditions,  including:  sunlight  and  nighttime  conditions,  heavy  and  light 
aircraft,  a  variety  of  body  positions  (sitting  in  the  cockpit,  under  the  wing,  in  a  wheel 
well,  in  a  crew  rest  bunk,  etc.),  varying  weather  conditions  from  extreme  summertime 
heat  to  near  freezing  temperatures  in  springtime.  While  the  HMD  was  “usable”  in  all 
these  environmental  conditions,  these  conditions  identified  areas  for  design  improvement. 

Usability  studies,  such  as  the  one  just  described,  have  provided  valuable  feedback 
about  one  wearable  computer  or  components  of  a  given  wearable  computer.  However, 
they  have  not  directly  compared  the  usability  of  a  wearable  computer  with  alternative 
portable  computers,  nor  have  they  compared  various  portable  computers  with  each  other. 
The  current  series  of  studies  addresses  these  gaps  in  usability  studies  to  date. 
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Method 


Five  hardware  configurations  were  compared  in  the  current  study.  The  FM-Net 
consisted  of  a  Motorola  handheld  radio  on  which  work  orders  were  opened  via  voice 
commands.  A  notebook  configuration  was  provided  through  the  Itronix  ruggedized 
computer.  The  Libretto  computer  was  a  miniature,  lightweight  computer.  Two 
configurations  of  the  Xybemaut  wearable  computer  were  tested:  one  included  a  Head- 
Mounted  display  and  the  other  included  a  wrist-worn  display  (see  Figure  3).  All 
computers  were  full  computers  running  MS  Windows. 


Portable  Radio 

Computers 


Libretto 


FM-Net 


Itronix 


Wearable 

Computers 


Xybemaut-Wrist 


Xybemaut-Head 


Figure  3.  Hardware  configurations 

For  scheduling  purposes,  a  total  of  4  hours  were  needed  to  test  a  participant. 
Inbrief  and  training  required  Vi  hour;  testing  required  no  more  than  2 Vi  hours;  the 
completion  of  Post-Test  Questionnaires  required  lA  hour.  Experimenters  needed  an 
additional  lA  hour  to  reconfigure  hardware  and  prepare  for  the  next  participant. 

Hardware  Requirements 

1 .  FM  Net  radio  with  converter  box 

2.  FM  net  workstation,  located  in  the  MOCC  for  receipt  of  radio  transactions 

3.  Itronix  ruggedized  notebook  computer  for  use  on  the  flightline 

4.  Libretto  miniature  notebook  computer  for  use  on  the  flightline 
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5.  Xybernaut  wearable  computer 

6.  Head-mounted  display  for  Xybernaut 

7.  Wrist-worn  keyboard  for  Xybernaut 

8.  Wrist-mounted  display  for  Xybernaut 

9.  Light  Meter 

10.  Sound  Meter 

Users 

Users  were  C-130  Talon  II  maintainers  qualified  to  open  work  orders.  These 
maintainers  included  Crew  Chiefs  and  Communication/Navigation  Specialists.  Ten  users 
participated  in  the  test:  5  Crew  Chiefs  and  5  Communication/Navigation  Specialists. 
Experience  level  was  not  controlled  in  the  study  and  ranged  from  inexperienced  (new  to 
the  career  field)  to  very  experienced  (dual  AFSC  qualified  and  many  years  in  the  career 
field). 

Facilities 

The  15th  Aircraft  Maintenance  Unit  (AMU)  located  at  Hurlburt  Field  served  as  the 
test  facility.  The  test  aircraft  was  the  MC-130H  Combat  Talon  II. 

Data  Collection  Team 

The  data  collection  team  consisted  of  four  individuals:  1)  a  subject  matter  expert, 
2)  a  videographer,  3)  a  subject  coordinator,  and  4)  a  flightline  coordinator.  The  subject 
matter  expert  provided  the  majority  of  the  interaction  with  each  Crew  Chief  and 
Specialist.  The  videographer  was  responsible  for  video  and  audio  documentation,  via 
camcorder,  of  each  experimental  session.  The  subject  coordinator  was  responsible  for 
scheduling  participant  testing  times  via  the  Hurlburt  POC,  and  arranging  and  carrying  out 
the  inbrief  and  outbriefs  sessions  of  each  participant.  The  flightline  coordinator  was 
responsible  for  ensuring  that  all  necessary  hardware  (e.g.,  computers)  was  available  and 
ready. 

Procedure 

Users  were  required  to  open/create  3  work  orders  using  the  FM  Net  radio,  Itronix 
computer.  Libretto  notebook  computer,  Xybernaut  head-mounted  display  configuration, 
and  the  Xybernaut  wrist-mounted  display  configuration  (Department  of  Defense,  2000). 
Only  the  third  work  order  for  a  given  equipment  configuration  was  observed  for  data 
collection.  Each  technician  performed  the  following  activities: 

1 .  Before  testing  the  subject  coordinator  conducted  an  inbrief  session  with  the 
participant.  This  inbrief  included  familiarization  with  the  software. 

2.  After  completion  of  the  inbriefing  the  flightline  coordinator  assisted  the 
participant  in  becoming  familiar  with  the  specific  equipment  configuration. 

3.  Once  the  participant  was  familiar  with  the  equipment  the  subject  matter  expert 
directed  the  participant  through  the  scenarios. 

4.  When  the  participant  had  finished  the  scenarios  for  a  given  equipment 
configuration  the  participant  completed  a  Post-Condition  Questionnaire. 
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5.  Upon  completion  of  the  Post-Condition  Questionnaire,  the  flightline  coordinator 
helped  the  participant  become  familiar  with  the  next  equipment  configuration. 

6.  This  process  continued  until  all  five  devices  had  been  tested. 

7.  When  each  participant  was  finished  with  the  hardware  testing  they  completed  a 
Post-Test  Questionnaire  and  received  an  outbrief  session  with  the  subject 
coordinator. 
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Results 


Both  objective  and  subjective  data  were  collected  during  the  Spiral  1  testing. 
Results  are  presented  for  each  type  of  data  collected. 

Objective  Data 

Completion  times  and  errors  were  collected  as  a  part  of  this  usability  test. 
However,  the  purpose  of  this  test  was  not  to  compare  the  devices  from  an  objective, 
performance-based  perspective.  The  purpose  was  to  compare  devices  from  a  subjective, 
usability-based  perspective.  The  objective  data  presented  in  this  section  are  included  for 
completeness.  The  Subjective  Data  section  provides  results  most  pertinent  to  the 
usability  test. 

Completion  Time 

For  each  device  the  participants  were  timed  from  the  initiation  to  the  submission 
of  opening/creating  a  work  order.  Figure  4  shows  the  average  completion  time  for  each 
device.  Clearly  the  FM-Net  condition  resulted  in  the  shortest  performance  times.  From 
an  empirical  standpoint  this  finding  should  not  be  overlooked.  While  statistical  measures 
of  comparison  were  not  made,  the  indication  is  that  the  voice  recognition  technology 
offered  through  the  FM-Net  condition  may  substantially  reduce  the  time  required  to  open 
work  orders  on  the  flightline. 
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Both 


Crew  Chief 


Comm/Nav 


Figure  4.  Completion  Times 

In  contrast,  the  Xybernaut  HMD  condition  resulted  in  the  longest  performance 
times.  Of  the  remaining  three  devices,  completion  times  were  within  approximately  1 0 
seconds  of  one  another. 

Errors 

Due  to  slight  differences  in  the  testing  scenarios,  the  results  for  the  FM  Net  trials 
are  discussed  separately  from  the  trials  for  the  other  devices. 

For  Itronix,  Libretto,  Xybernaut  HMD  and  Xybernaut  Wrist,  eleven  data  entry 
errors  were  found  in  the  test  trials.  Nine  trials  were  in  error  because  the  WD  code  (When 
Discovered  code)  field  was  not  populated.  The  two  remaining  errors  were  contained  in 
the  WCE  Narrative  field.  Of  these  eleven  errors,  eight  were  associated  with  the 
Xybernaut  devices  (4  each  with  Xybernaut  HMD  and  Xybernaut  Wrist).  Two  errors 
were  associated  with  the  Libretto,  and  one  with  Itronix  (Figure  5).  Not  included  in  this 
total  are  the  two  errors  discussed  below  for  FM  Net. 

Of  the  eleven  errors  discussed  above,  Comm/Nav  Specialists  recorded  seven, 
while  only  four  were  recorded  by  Crew  Chiefs. 

Two  errors  were  recorded  for  the  FM  Net  trials,  both  due  to  inaccurate  field 
population  (one  each  in  Aircraft  #  and  System).  Comm/Nav  Specialists  committed  both 
of  these  errors.  Crew  Chiefs  did  not  commit  any  errors  during  FM  Net  trials. 
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In  six  of  the  ten  FM  Net  trials,  subjects  did  not  populate  the  Discrepancy  field. 
While  this  was  not  considered  an  error  for  this  particular  test  (as  long  as  the  System  field 
was  populated),  in  the  future,  there  should  be  some  visual  feedback  to  users  about  the  fact 
that  the  System  information  they  are  verbalizing  is  being  used  to  fill  in  the  Discrepancy 
field. 

Note  that  the  intent  of  this  usability  test  was  not  to  perform  empirical  testing  (i.e., 
statistical  analysis).  Therefore  due  to  the  design  of  the  usability  test,  an  insufficient 
number  of  subjects  were  available  for  empirical  analysis  of  these  error  data.  These  data 
are  included  in  the  report  for  the  purpose  of  helping  to  identify  potential  areas  for  future 
empirical  tests. 


Errors  by  Device 


Itronix  FM  Net  Libretto  XYHead  XYWrist 


Figure  5.  Errors  by  Device 


Subjective  Data 


Post-Condition  Questionnaires 

The  primary  question  for  the  Post-Condition  Questionnaires  dealt  with  the  ease  in 
opening  work  orders.  This  scaled  question  was  asked  following  use  of  each  type  of 
hardware.  End  point  labels  (“easy”  and  “not  at  all  easy”)  were  selected  based  on 
recommendations  by  Dumas  (1998)  on  how  to  get  the  most  diagnostic  feedback  from 
users.  This  question  was  on  a  5-point  Likert  scale: 


Easy 

Not  at  all  easy 

1 

2 

3 

4 

5 

A  cluster  graph  for  the  question  is  provided  below  to  illustrate  the  responses 
provided  by  each  participant.  Interpretation  of  the  graph  is  as  follows: 
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1 .  Usability  problems  and  potential  problems  are  apparent  only  if  two  or  more 
participants  give  ratings  of  3,  4  or  5. 

2.  If  one  or  more  participants  give  a  rating  of  3,  and  only  one  participant  gives  a 
rating  of  4  or  5  then  a  potential  usability  problem  is  indicated. 

3.  If  two  or  more  participants  give  a  rating  of  4  or  5,  a  definite  usability  problem 
is  indicated. 

Using  this  set  of  guidelines,  Figure  6  indicates  that  no  usability  problems  exist  for 
ease  of  use  for  the  Itronix,  Libretto,  or  FM-Net  devices;  however,  users  did  seem  to  have 
potential  ease  of  use  problems  with  the  Xybemaut  Wrist  device,  and  problems  with  the 
Xybemaut  Head-Mounted  Display  (two  or  more  users  indicated  a  4  or  5). 


Ease  of  Use  Ratings 


Easy  Not  at  all 

Easy 

Figure  6.  Post-Condition  Ease  of  Use  Rating  for  all  5  equipment  configurations 

In  usability  evaluations,  the  trends  or  patterns  of  responses  must  be  carefully 
considered.  In  this  case,  for  the  FM-Net  and  Itronix  conditions  only  one  person  rated  the 
device  a  3  (middle  of  the  scale);  however,  for  the  Xybemaut  Wrist  condition  two  rated  it 
a  3  and  there  were  an  even  number  that  rated  it  as  a  1  or  2.  This  indicates  a  small 
negative  shift  in  usability  in  comparison  with  the  other  three  devices  -  a  potential 
problem  with  usability.  The  Xybemaut  HMD  condition  indicates  even  a  further  negative 
shift  in  usability,  with  two  people  rating  the  device  toward  the  not  at  all  easy  to  use  range. 

Comments  from  users  further  substantiate  these  results.  Users  indicated  that  for 
the  Xybemaut  HMD  condition  the  equipment  was  too  bulky  and  got  in  the  way  of 
moving  around  in  tight  places.  They  also  indicated  the  equipment  needed  to  be  cordless, 
and  that  the  screen  was  not  visible  in  sunlight.  These  are  all  issues  that  substantially 
reduce  the  usability  of  a  device.  With  respect  to  the  Xybemaut  Wrist  condition,  users 
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gave  similar  comments.  Users  did  not  like  the  bulkiness  or  cords;  however,  they  did  like 
the  screen  size  for  the  wrist-mounted  display  and  they  liked  the  touch  screen  offered  by 
this  display. 

Issues  identified  by  users  with  respect  to  the  other  three  devices  were  milder. 
They  found  the  Itronix  device  too  heavy  and  the  FM-Net  vocabulary  somewhat  limited; 
however,  these  somewhat  negative  comments  were  overshadowed  by  the  positive 
comments  given  about  these  devices.  For  the  Libretto  and  FM-Net  conditions,  users 
specifically  stated  that  they  were  easy  to  use.  They  liked  the  fact  that  these  two  devices 
were  lightweight  and  had  a  small  footprint.  They  also  indicated  that  they  liked  the  screen 
readability  and  the  touch  screen  capability  of  the  Itronix  device  (this  is  consistent  with 
their  positive  comments  about  the  Xybemaut  wrist-mounted  display). 

Post-Test  Questionnaires 

The  Post-Test  Questionnaires  addressed  the  participants’  preference  of  equipment 
configuration  (see  Appendix  B  for  a  sample  of  questionnaires  administered).  The 
participants  rank  ordered  the  devices  on  4  criteria:  1)  display,  2)  text  entry,  3)  field 
selection,  4)  and  overall  technology.  The  following  data  are  averages  of  the  participants’ 
rank  orders  (see  Figures  7-10). 

Cluster  graphs  are  provided  below  to  illustrate  the  rankings  provided  by  each 
participant.  Interpretation  of  the  graph  is  as  follows: 

1 .  Clusters  that  are  evenly  distributed  from  most  preferred  to  least  preferred  for 
two  or  more  hardware  devices  indicate  no  preference. 

2.  Clusters  that  are  unevenly  distributed  indicate  that  some  preference  is 
observed. 

Using  this  set  of  guidelines,  Figure  7  indicates  that  there  are  no  preferences 
between  the  Libretto  and  Itronix  devices  for  display  (even  distribution).  However,  the 
uneven  distribution  of  ratings  for  the  Xybemaut  Wrist  and  HMD  devices  indicates  that 
the  Xybemaut  HMD  is  the  least  preferred  while  the  Xybemaut  Wrist  is  the  second  least 
preferred. 
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Display  Rankings 


Figure  7.  Ranking  for  Display 

Note  that  the  FM-Net  condition  was  a  voice  only  system  (without  any  display 
device);  therefore,  it  was  not  included  in  this  rank  order  comparison. 

Ranking  of  the  devices  for  display  clearly  shows  that  the  Xybemaut  HMD  was 
the  least  preferred  condition.  Technicians  noted  that  this  monocular  head-mounted 
display  device  was  difficult  to  see  in  bright  sunlight.  Many  technicians  had  to  readjust 
the  position  of  the  eyepiece  while  wearing  it.  It  should  also  be  noted  that  the  study  was 
conducted  in  Florida,  in  July.  Reported  heat  index  (combined  heat  and  humidity)  was 
between  90  and  110  each  day.  Flightline  conditions  were  probably  worse  than  this. 
While  technicians  did  not  indicate  that  the  heat  had  an  effect  on  their  rankings, 
experimenters  were  required  to  clean  the  head-mounted  display  after  each  test  to  remove 
perspiration  from  the  device.  Other  laboratory  head-mounted  display  experiments  have 
also  shown  this  to  be  problematic  with  users  (Unger,  Quill  &  Masquelier,  1998).  With 
respect  to  the  other  devices  technicians  commented  that  they  liked  the  size  of  the 
Xybemaut-wrist  mounted  display,  although  it  was  ranked  second  to  last  for  preference. 
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Text  Entry  Rankings 


H  Comm/Nav 
□  Crew  Chief 


Figure  8.  Ranking  for  Text  Entry 

For  text  entry  the  device  least  preferred  was  the  Xybemaut  with  the  wrist- 
mounted  display.  The  Xybemaut  Wrist  condition  had  an  on-screen  keyboard  and 
technicians  were  required  to  move  the  keyboard  around  the  screen  to  view  the  fields 
being  filled  in.  As  shown  in  the  figure,  they  did  not  care  for  the  on-screen  keyboard  as 
presented.  Several  participants  indicated  that  if  an  on-screen  keyboard  is  to  be  used,  it 
should  be  fixed  at  the  bottom  of  the  screen  and  not  cover  any  of  the  material  presented  on 
the  work  order. 

The  Xybemaut  HMD  provided  a  wrist-worn  keyboard.  User’s  ranked  the  wrist 
keyboard  (on  the  Xybemaut  HMD  condition)  slightly  better;  however  text  entry  and  field 
selection  frequently  intermixed.  For  the  Xybemaut  HMD  condition,  technicians  were 
required  to  select  a  field  with  the  hip-mounted  pointing  device  and  then  fill  in  the  text 
with  the  wrist-wom  keyboard.  Experimenters  observed  that  this  right-hand  movement 
from  the  right  hip  to  the  left  arm  keyboard  was,  at  times  quite  awkward  for  the 
technicians.  This  may  have  added  to  the  relatively  poor  ranking  given  to  this  device  for 
text  entry. 

For  text  entry,  the  Itronix  device  and  the  Libretto  offered  a  full  keyboard.  It  is 
interesting  that  users  did  not  identify  a  preference  for  verbal  text  entry  (offered  by  the 
FM-Net  condition)  over  the  keyboard  entry.  Entering  text  verbally  is  very  intuitive,  and 
several  technicians  commented  that  it  was  very  easy  to  use.  However,  other  technicians 
expressed  difficulty  with  entering  text  in  the  current  application.  Two  concerns  were 
raised:  technicians  wanted  more  flexibility  in  the  vocabulary  accepted  by  the  system 
(i.e.,  an  expanded  vocabulary),  and  they  wanted  a  visual  reference  to  what  had  been 
entered. 
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Field  Selection  Rankings 


FM  Net 


I 

w. 
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Libretto 
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Itronix 


Xy  Wrist 


Xy  HMD 
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Most  Preferred 


IH  Comm/Nav 
□  Crew  Chief 


□ 


□ 


Least  Preferred 


Figure  9.  Ranking  for  Field  Selection 

Technicians  preferred  the  FM-Net  voice  recognition  system  to  the  other  devices. 
For  the  FM-Net  condition,  a  crib  sheet  was  attached  to  each  radio.  This  provided  users 
with  a  visual  reference  as  to  what  fields  needed  to  be  verbally  selected.  This  visual 
reference  along  with  the  ease  of  verbal  input  made  this  selection  easy  to  users. 
Interestingly,  the  Xybemaut  Wrist  condition  was  ranked  second  to  the  FM-Net  condition. 
This  device  provided  a  touch  screen  for  field  selection  and  was  mounted  on  the  left  wrist 
(all  subjects  were  right  handed).  This  configuration  appears  to  have  been  somewhat 
conducive  to  field  selection. 

The  Xybemaut  HMD  condition  with  the  hip-mounted  mouse  was  ranked  last  in 
terms  of  field  selection.  Again,  this  appeared  to  be  due  to  the  interaction  of  the  hip- 
mounted  mouse  and  the  wrist-mounted  keyboard. 
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Figure  10.  Ranking  for  Overall  Technology 

Technicians  preferred  the  Libretto  computer  to  the  other  devices.  This  is  true  for 
both  Crew  Chiefs  and  Comm/Nav  Specialists.  Technicians  indicated  that  this 
lightweight,  small  device  was  very  easy  to  use.  Clearly  this  ranking  indicates  their 
preference  for  the  device.  It  is  noteworthy  that  the  Comm/Nav  Specialists’  rankings 
indicate  that  they  also  seemed  to  like  the  FM-Net  and  Itronix  devices.  The  least  preferred 
device  was  the  wearable  Xybemaut  computer  with  the  head-mounted  display.  The 
Xybemaut  computer  with  the  wrist-mounted  display  was  ranked  second  to  last. 
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Discussion 


FM-Net 

The  Completion  Time  data  showed  that  the  users  took  the  least  amount  of  time  to 
complete  the  work  orders  with  the  FM  Net  radio.  According  to  the  Post-Condition 
Questionnaire  the  FM  Net  radio  was  rated  as  easy  to  use.  The  data  from  the  Post-Test 
Questionnaire  indicated  that  the  FM-Net  radio  was  the  preferred  device  for  selecting 
fields  on  the  screen.  Technicians  ranked  the  FM-Net  equivalent  to  the  Itronix  and 
Libretto  devices  for  text  entry.  Users  commented  that  they  did  like  the  lightweight  nature 
and  portability  of  the  radio. 

An  interesting  finding  was  that  some  users  did  not  prefer  the  FM-Net  for  entering 
text  (an  intuitively  obvious  choice) — instead  they  found  it  equivalent  to  entering  text  with 
the  Itronix  and  Libretto  keyboards.  Users  expressed  difficultly  using  a  specific 
vocabulary  in  order  to  operate  the  radio.  Another  problematic  issue  was  that  the  radio  did 
not  provide  the  user  with  any  type  of  feedback  or  confirmation  that  the  work  order  had 
been  received.  Users  may  have  given  the  FM  Net  radio  a  better  ranking  if  it  provided 
some  sort  of  visual  feedback.  That  feedback  could  appear  on  a  computer  screen  or  be 
read  back  to  the  user  via  the  radio.  Based  on  technician’s  familiarity  with  radios  the  FM- 
Net  radio  could  be  the  device  of  choice  if  these  features  could  be  incorporated  into  it. 

Itronix  Notebook 

Results  from  the  Post-Condition  Questionnaire  showed  that  technicians  found  the 
Itronix  computer  as  easy-to-use.  Data  from  the  Post-Test  Questionnaire  showed  that 
technicians  preferred  the  Itronix  display  (equivalent  ranking  was  given  to  the  Libretto). 
They  also  preferred  the  Itronix  computer  for  text  entry  (equivalent  rankings  were  given  to 
the  Libretto  and  the  FM-Net).  Users  commented  that  the  computer’s  touch  screen  and 
stylus  pen  did  aid  them  with  maneuvering  throughout  the  screen.  The  users  also  liked  the 
ruggedized  casing  for  the  Itronix  computer.  Because  the  computer  was  ruggedized,  the 
users  did  not  feel  the  need  to  be  as  cautious  with  it  as  they  were  with  the  other  equipment 
configurations. 

The  weight  of  the  Itronix  computer  proved  to  be  a  problem.  The  users 
commented  that  the  computer  was  too  heavy  or  “bulky”.  Because  of  its  ruggedization  the 
Itronix  computer  was  heavier  than  a  typical  notebook  computer.  If  the  Itronix  computer 
was  smaller  and  lighter  users  might  rate  it  higher. 

Libretto  Miniature  Computer 

According  to  the  Post-Condition  Questionnaire,  the  Libretto  was  rated  as  easy-to- 
use — with  all  subjects  rating  it  with  a  1  or  2.  Data  from  the  Post-Test  Questionnaire 
showed  that  technicians  preferred  the  Libretto  display  (equivalent  ranking  was  given  to 
the  Itronix  computer).  They  also  preferred  the  Libretto  computer  for  text  entry 
(equivalent  rankings  were  given  to  the  Itronix  and  the  FM-Net).  Post-Test  Questionnaire 
data  revealed  that  the  Libretto  computer  was  rated  the  preferred  technology  overall. 

Users  liked  the  size  of  the  computer,  and  they  liked  that  the  Libretto  was  small  enough  to 
carry  in  their  hand. 
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Some  users  indicated  that  the  Libretto  should  be  ruggedized.  Another  noted 
problem  was  with  the  sensitivity  of  the  mouse-like  device  used  for  field  selection.  Users 
felt  it  moved  too  quickly  around  the  screen. 

Xvbernaut:  Head  Mounted  Display  Configuration 

The  Post-Condition  Questionnaire  data  showed  that  this  configuration  of  the 
Xybemaut  computer  was  the  most  difficult  device  to  use.  Two  technicians  rated  ease-of- 
use  in  categories  previously  defined  as  unacceptable.  That  is,  the  Usability  Test  Plan 
clearly  identified  that  ratings  of  4  and  5  were  unacceptable.  The  Post-Test  Questionnaire 
data  illustrated  that  the  head-mounted  configuration  was  the  least  preferred  display,  field 
selection  device,  and  technology  overall.  Users  commented  that  the  various  computer 
cables  made  it  difficult  to  move  around.  The  weight  of  the  device  also  proved  to  be  a 
problem  for  the  users.  Several  users  commented  that  the  configuration  was  too  “bulky.” 
As  for  the  display,  users  had  difficulty  with  sunlight  glare  on  the  HMD.  They  also  had 
difficulties  adjusting  the  display’s  eyepiece  so  that  the  entire  screen  would  be  in  focus. 

Xvbernaut:  Wrist  Mounted  Display  Configuration 

The  Post-Condition  Questionnaire  data  showed  the  wrist-mounted  configuration 
was  easy-to-use;  however,  ratings  were  relatively  low  in  comparison  with  ratings  given  to 
the  other  devices.  The  Post-Test  Questionnaire  data  showed  that  the  wrist-mounted 
configuration  was  ranked  second  to  the  FM-Net  condition  for  field  selection.  The  touch 
screen  capability  offered  by  this  device  appears  to  be  conducive  to  field  selection.  Users 
indicated  that  they  liked  the  touch  screen  nature  of  the  wrist  display  and  the  use  of  the 
stylus  pen. 

This  device  was  ranked  second  to  last  for  technology  overall.  Comments 
provided  by  users  were  similar  to  the  comments  made  for  the  head-mounted  display 
configuration.  Technicians  did  not  like  the  weight  and  “bulkiness”  of  the  computer  and 
display.  The  cables  made  it  difficult  to  move  around. 
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Conclusions 


This  test  compared  five  portable  computing  devices  used  for  opening  work  orders 
on  the  flightline.  Air  Force  Crew  Chiefs  and  Communication/Navigations  Specialists  for 
the  MC-130H  aircraft  participated  in  the  study.  These  technicians  identified  that,  among 
the  devices  shown,  the  Libretto  computer  was:  1)  rated  highest  for  overall  ease-of-use, 
and  2)  the  preferred  technology  overall.  This  device  is  small  and  lightweight;  it  offers  a 
full  keyboard  and  full  screen  that  technicians  can  carry  in  one  hand. 

Features  of  the  other  devices  should  be  considered  in  addition  to  those  of  the 
Libretto.  The  ruggedization  of  the  Itronix  device  was  seen  as  a  benefit  to  technicians. 
Also,  the  touch  screen  offered  by  the  Itronix  and  Xybernaut  Wrist  conditions  was  a 
desirable  feature  for  selecting  fields;  however,  for  manual  text  entry  it  should  be  noted 
that  physical  keyboard  is  still  preferred  to  an  on-screen  keyboard. 

While  the  Libretto  computer  was  the  preferred  device  in  this  test,  the  potential  for 
the  FM-Net  cannot  be  overstated.  Timesavings  offered  by  FM-Net  are  potentially 
substantial.  According  to  this  report,  usability  issues  associated  with  this  device  were 
limited  to  providing  feedback  to  the  user  on  the  text  just  entered  (e.g.,  visual  feedback) 
and  expanding  the  vocabularies  used.  Should  these  usability  issues  be  addressed,  the 
potential  exists  for  a  device  that  offers  not  only  performance  improvement  but  also 
usability  acceptance  by  aircraft  flightline  maintainers. 

While  previous  studies  conducted  by  the  Air  Force  Research  Laboratory  have 
indicated  that  better  performance  is  obtained  when  using  a  wearable  computing  system, 
usability  comparison  of  these  devices  with  portable  computers  clearly  indicates  that 
wearable  systems  are  not  as  usable  in  conditions  such  as  opening  work  orders  from  the 
flightline.  Preliminary  error  data  indicated  that  more  empirical  testing  is  also  required  on 
errors  committed  when  using  an  HMD  for  opening  work  orders  on  the  flightline.  If  both 
improved  performance  and  increased  usability  are  to  be  considered  for  future  systems, 
wearable  computer  research  must  not  only  look  at  empirical  testing,  but  should  focus  on 
the  usability  issues  identified  in  this  study. 
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Method 


Ten  maintenance  technical  data  layouts  were  presented  on  three  portable 
maintenance  aids.  The  Panasonic  Toughbook  34  (full  screen),  LXE-MX3  (half  screen), 
and  Intermec  710  (quarter  screen)  were  the  devices  used  to  display  the  technical  data. 

The  Toughbook  was  a  ruggedized  computer,  and  the  LXE-MX3  and  the  Intermec  710 
were  both  ruggedized  handheld  computers.  The  Toughbook  used  Windows  NT  operating 
system,  and  Adobe  Acrobat  Reader  program  to  display  the  technical  data.  The  LXE- 
MX3  used  Windows  CE  operating  system,  and  Ansyr  Primer  3.1  program  to  display  the 
technical  data.  The  Intermec  710  used  Windows  Pocket  PC  operating  system,  and  Ansyr 
Primer  3.1  program  to  display  the  technical  data  (see  Figure  1 1). 


Panasonic 

Toughbook 


Full  Screen 


Half  Screen  Quarter  Screen 


Figure  11.  Hardware  platforms  (screens) 

For  scheduling  purposes,  a  total  of  314  hours  were  needed  to  run  each  subject.  Three 
technicians  were  expected  to  perform  the  test  at  the  same  time.  Inbrief  and  training 
required  !4  hour;  testing  required  no  more  than  214  hours;  post-test  discussion  among  the 
participants  and  experimenters  required  14  hour. 

Hardware  Requirements 

1.  Panasonic  Toughbook  34  series  with  option  of  192M  RAM,  spare  battery,  and 
charger 

2.  LXE-  MX3  handheld  computer  with  16M  RAM,  spare  battery,  and  charger 

3.  Intermec  handheld  computer  with  16M  RAM,  710  series,  spare  battery,  and  charger 

4.  At  least  one  (1)  audio  recorder  with  batteries 

5.  Blank  videocassette  tapes  (14)  or  audio  tapes 
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6.  Power  Strips  (1) 

7.  Extension  Cords  (1) 

8.  Clipboards  (3) 

9 .  Package  of  pens  ( 1 ) 

10.  Digital  Camera 

Users 

Users  were  C-130  Talon  II  maintainers  qualified  to  use  technical  data  on  the 
Flightline.  Fifteen  users  participated  in  each  module  of  the  test.  Technicians  included 
Crew  Chiefs  and  Specialists. 

Facilities 

The  C-130  15th  Aircraft  Maintenance  Unit  (AMU)  located  at  Hurlburt  Field 
served  as  the  test  facility.  Testing  occurred  in  the  unit  conference  room. 

Data  Collection  Team 

The  data  collection  team  consisted  of  three  persons:  a  subject  matter  expert 
(SME)  and  two  experimenters.  The  subject  matter  expert  provided  knowledge  in 
assuring  that  technicians  understood  the  purpose  of  the  test  as  it  relates  to  their  daily  job. 
Experimenters  ensured  that  subjects  understood  the  requirements  of  the  test  and  that 
appropriate  data  was  being  collected  during  the  effort. 

Procedure 

The  purpose  of  the  test  was  to  determine  the  usability  of  technical  data  given  the 
varying  screen  sizes  available  on  the  three  devices.  Users  were  required  to  view  1 0  types 
of  technical  data  layouts: 

Data  Format  1 :  06  Manual  (e.g.,  work  unit  codes) 

Data  Format  2:  Job  Guides 

Data  Format  3 :  General  Service  manual  descriptions 

Data  Format  4:  Work  Cards 

Data  Format  5:  Fault  Isolation  Trees 

Data  Format  6:  Fault  Reporting  Manuals  (including  diagrams  turned  90°) 

Data  Format  7:  General  System  diagrams  with  continuation  pages 
Data  Format  8:  Schematic  diagrams  (from  814  x  11  inch  pages)  with  continuation 
pages  turned  90° 

Data  Format  9:  Schematic  diagrams  (from  11x17  inch  pages)  turned  90° 

Data  Format  10:  Schematic  diagrams  (from  11x17  inch  pages)  with  continuation 
pages  turned  90° 

Three  technicians  participated  in  each  session.  Each  technician  following  the 
same  set  of  procedures  as  follows: 

1 .  After  completing  the  training,  the  three  technicians  were  given  one  of  the 
three  portable  devices  (full  screen,  half  screen,  or  quarter  screen). 

2.  They  were  then  instructed  to  select  a  presentation  layout. 

3.  Once  the  layout  had  been  selected  each  participant  was  instructed  to  make  the 
information  viewable  on  the  screen  (by  zooming  or  changing  the  page  layout). 
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4.  After  viewing  the  presentation  layout  the  participant  responded  to  three 
questions  on  the  post-condition  questionnaire  (see  Appendix  C  for  a  sample  of 
questionnaires  administered). 

5.  Participants  then  moved  on  to  the  next  presentation  layout  and  repeated  the 
two  previous  steps. 

6.  Following  the  last  layout  technicians  switched  devices  and  repeated  the  four 
previous  steps. 

7.  After  each  technician  had  used  all  three  devices  there  was  a  discussion  among 
the  technicians,  the  SME,  and  the  experimenters.  The  discussion  focused  on 
problems  the  technicians  had  with  each  of  the  devices. 
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Results 


Subjective  data  were  collected  during  the  Spiral  3  Synthetic  testing.  Due  to  the 
increased  number  of  subjects  available  and  the  controlled  setting,  statistical  analysis  was 
possible  for  these  data. 

Post-Condition  Questionnaires 


Twenty- four  questions  were  asked  following  use  of  each  type  of  hardware.  These 
questions  were  on  a  5-point  Likert  scale: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

These  five  response  alternatives  were  selected  based  on  evidence  that  they  are  at 
least  one  standard  deviation  apart  from  one  another  and  have  parallel  wording  (Babbitt,  & 
Nystrom,  1989).  A  cluster  graph  for  each  question  is  provided  below  to  illustrate  the 
responses  provided  by  each  participant.  The  method  for  interpreting  the  cluster  graph 
was  established  a  priori  to  the  test  and  is  as  follows: 

1.  Usability  problems  and  potential  problems  are  apparent  only  if  two  or  more 
participants  give  ratings  of  3,  4  or  5. 

2.  If  one  or  more  participants  give  a  rating  of  3,  and  only  one  participant  gives  a 
rating  of  4  or  5  then  a  potential  usability  problem  is  indicated. 

3.  If  two  or  more  participants  give  a  rating  of  4  or  5,  a  definite  usability  problem 
is  indicated. 

Using  this  set  of  guidelines,  Figure  12  indicates  that  no  usability  problems  exist 
for  reading  technical  data  on  the  Toughbook;  however,  users  identified  problems  reading 
the  LXE  and  Intermec  devices  (two  or  more  users  indicated  a  4  or  5). 

Subjective  Data 

For  each  of  the  three  devices  subjects  responded  to  twenty- four  questions 
concerning  the  use  of  technical  data.  Questions  related  to  the  10  types  of  technical  data 
layouts.  A  wide  range  of  technical  data  was  used  to  determine  which  types  might  be 
appropriate  for  flightline  use.  While  data  were  collected  on  the  various  types  of  technical 
data,  only  questions  3, 4,  and  5  are  analyzed,  in  depth,  for  the  Synthetic  Usability  Test 
data.  These  three  questions  relate  specifically  to  use  of  a  Job  Guide  procedures,  which 
was  the  type  of  technical  data  selected  for  the  subsequent  Field  Usability  Test.  The 
statistical  results  for  all  twenty-four  questions  are  presented  in  Appendix  D. 

Question  3:  Ease  of  Reading 

For  question  3,  there  was  a  significant  difference  between  the  participants’ 
response  for  the  Toughbook  and  their  response  for  the  other  two  devices.  A  single  factor 
ANOVA  showed  significant  differences  (F(2,  42)=6.402,  p<.05).  In  general,  the 
participants  believed  it  was  easier  to  read  the  technical  data  displayed  on  the  Toughbook 
display  than  the  LXE  and  Intermec  displays.  The  participants  experienced  some 
problems  reading  the  information  on  the  LXE  and  Intermec  displays  (see  Figure  13). 
These  findings  are  consistent  with  those  findings  shown  in  the  cluster  graph  on  Figure  12. 
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Job  Guides 


Figure  12.  Ease  of  Reading 


Job  Guides 


Figure  13.  Ease  of  Reading  average  ratings 
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Question  4:  Ease  of  Finding  Associated  Graphic 

A  single  factor  ANOVA  showed  that  there  was  a  statistically  significant 
difference  between  the  participants’  response  for  the  Toughbook  and  their  response  for 
the  other  two  devices  F(2,  42)=3.897,  p<.05.  In  general,  the  participants  believed  it  was 
easier  to  find  graphics  with  the  Toughbook  display  than  with  the  LXE  or  Intermec 
displays.  Inspection  of  the  cluster  graph  shows  that  users  found  the  Toughbook 
Completely  Acceptable.  This  graph  also  indicates  potential  usability  problems  with  the 
Intermec  for  finding  associated  graphics  (see  Figures  14  &  15). 
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Job  Guides 


Question  4:  Ease  in  finding  associated  graphic 
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Figure  14.  Ease  of  Finding  Associated  Graphic 
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Question  4:  Ease  in  finding  associated  graphic 
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Figure  15.  Ease  of  Finding  Associated  Graphic  average  ratings 
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Question  5:  Ability  to  associate  the  numerical  reference  points  between  the  graphic  and 
text 

A  single  factor  ANOVA  showed  that  there  was  a  statistically  significant 
difference  between  the  participants’  response  for  the  Toughbook  and  their  responses  for 
the  other  two  devices  F(2,  42)=6.618,  p<.05.  In  general,  the  participants  believed  it  was 
easier  to  associate  numerical  reference  points  between  the  graphics  and  text  with  the 
Toughbook  display  than  with  the  LXE  or  Intermec  displays.  Responses  plotted  in  the 
cluster  graph  indicate  that  the  participants  experienced  some  problems  with  the  LXE 
display,  and  that  they  had  some  potential  problems  in  associating  the  reference  points 
with  the  Intermec  display  (see  Figures  16  &  17). 
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Question  5;  Ability  to  associate  the  numerical 
reference  points  between  the  graphic  and  text 
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Figure  16.  Ability  to  associate  the  numerical  reference  points  between  the  graphic  and 

text 


Job  Guides 


Question  5:  Ability  to  associate  the  numerical  reference 
points  between  the  graphic  and  text 

Statistically  significant 
difference  between 
Toughbook  response  and 
other  two  responses. 


Toughbook 


Intermec 


Figure  17.  Ability  to  associate  the  numerical  reference  points  between  the  graphic  and 

text  average  ratings 
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Discussion 


Toughbook 

In  general,  the  results  indicated  the  participants  found  it  easiest  to  view  the 
Power-On  Job  Guide  on  the  Toughbook  display.  The  Toughbook  provided  a  larger  and 
better  quality  display  than  the  other  two  devices,  enabling  participants  to  easily  read  the 
technical  data  on  the  display.  The  size  and  quality  of  the  display,  along  with  the  touch 
screen  capability,  made  locating  specific  graphics  and  relating  them  with  the  appropriate 
infonnation  within  the  text  easier  for  the  participants.  The  touch  screen  capability 
allowed  the  participants  to  easily  maneuver  through  and  manipulate  the  technical  data 
that  was  displayed  (see  Figure  1 8). 


Figure  18.  Panasonic  Toughbook 


LXE 

As  indicated  by  the  results,  the  participants  had  some  difficulties  with  the  LXE 
device.  The  three  key  problems  dealt  with:  1)  the  readability  of  the  display,  2)  the 
sensitivity  of  the  touch  screen,  and  3)  the  speed  of  the  computer.  Poor  display  quality 
made  viewing  the  technical  data  difficult  for  participants.  The  touch  screen  capability 
was  not  sensitive  enough  for  the  participants,  and  because  of  the  sensitivity  issue  the 
participants  found  it  difficult  to  manipulate  the  information  on  screen.  Associating  the 
numerical  reference  points  between  the  text  and  graphics  was  somewhat  difficult  for  the 
participants.  Finally,  the  participants  found  the  processing  speed  of  the  LXE  to  be  a 
problem.  When  the  participant  attempted  to  move  to  the  next  page  or  view  a  graphic  they 
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would  have  to  wait  several  seconds  for  the  computer  to  load  that  page  or  graphic  (see 
Figure  19). 


Figure  19.  Using  the  LXE 


Intermec 

According  to  the  results,  participants  had  some  difficulties  with  the  display  on  the 
Intermec.  Specifically  the  participants  had  problems  with  the  readability  of  the  display. 
The  participants  found  that  it  was  difficult  to  read  larger  documents  (e.g.,  schematics)  on 
the  Intermec’s  quarter  screen  display.  This  issue  had  some  influence  on  the  participants’ 
ability  to  locate  specific  graphics  and  the  ability  to  associate  numerical  reference  points 
between  the  graphic  and  text.  It  should  be  noted  that  the  participants  preferred  the  overall 
size  of  the  Intermec  device  to  the  other  two  devices  (see  Figure  20). 
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Conclusions 


The  purpose  of  this  test  was  to  determine  the  usability  of  Job  Guides  given  the 
various  sizes  available  on  the  Panasonic  Toughbook  (full  screen),  LXE  (half  screen),  and 
Intermec  710  (quarter  screen).  Other  types  of  technical  data  were  included  in  the 
evaluation;  however,  Job  Guide  data  were  analyzed  in  detail  for  their  potential  use  on  the 
flightline.  Air  Force  Crew  Chiefs  and  Specialists  for  the  C-130  Talon  II  took  part  in  the 
study.  These  technicians  indicated  that  the  display  for  the  Panasonic  Toughbook  was 
preferred  overall.  The  size  and  quality  of  the  Panasonic  Toughbook  allowed  the 
participants  to  view  the  Job  Guides  more  easily.  The  touch  screen  capability  provided 
the  participants  with  an  easy  way  to  manipulate  the  Job  Guides. 

In  this  usability  test  the  LXE  and  Intermec  710  devices  were  not  rated  as  well  as 
the  Panasonic  Toughbook,  however,  both  of  these  devices  were  considered  to  be  within 
acceptable  limits  for  Spiral  3  Field  test  selection.  Participants  indicated  that  the  quality 
of  the  displays  for  these  two  devices  made  it  difficult  to  read  the  information  that  was 
presented.  It  was  also  determined  that  the  touch  screen  capability  of  the  LXE  device  was 
not  sensitive  enough  for  the  participants.  This  added  to  the  participants’  difficulty  in 
manipulating  the  Job  Guides.  Participants  indicated  that  if  these  issues  were  remedied, 
the  LXE  would  be  a  more  acceptable  device.  Finally,  the  participants  indicated  that 
although  the  display  image  quality  for  the  Panasonic  Toughbook  was  best,  the  size  of  the 
Intermec  710  was  more  desirable.  It  was  noted  by  several  technicians  that  if  the  quality 
of  the  Intermec’s  display  were  improved  it  would  be  a  more  acceptable  device. 
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Method 


Three  hardware  configurations  were  compared  in  the  current  study.  The 
Panasonic  Toughbook  provided  a  full  screen  display  in  a  ruggedized  notebook-like 
computer.  The  LXE  provided  a  half  screen  display  in  a  ruggedized  housing.  The 
Intermec  710  displayed  a  quarter  screen  image  in  a  ruggedized  housing  (see  Figure  21). 
The  Toughbook  computer  was  a  full  computer  running  MS  Windows  OS.  The  LXE’s 
operating  system  was  Windows  CD  and  the  Intermec  710  used  the  Pocket  PC  operating 
system. 


Panasonic 

Toughbook 


LXE  Intermec  710 


Full  Screen 


Half  Screen  Quarter  Screen 


Figure  21.  Hardware  configurations 

For  scheduling  purposes,  a  total  of  3  Vi  hours  was  needed  to  test  each  participant. 
Inbrief  and  training  required  Vi  hour;  testing  required  no  more  than  2Vi  hours;  the 
completion  of  Post-Test  Questionnaires  required  Vi  hour.  Experimenters  needed  an 
additional  Vi  hour  to  reconfigure  hardware  and  prepare  for  the  next  participant. 

Hardware  Requirements 

1 .  Panasonic  Toughbook  34  series  with  option  of  1 92M  RAM,  spare  battery,  and 
charger 

2.  LXE-  MX3  handheld  computer  with  16M  RAM,  spare  battery,  and  charger 

3.  Intermec  handheld  computer  with  16M  RAM,  710  series,  spare  battery,  and  charger 

4.  At  least  one  (1)  video  tape  recorder  with  batteries 

5.  Blank  videocassette  tapes  (14)  or  audio  tapes 

6.  Power  Strips  (1) 

7.  Extension  Cords  (1) 
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8.  Clipboards  (3) 

9.  Package  of  pens  (1) 

10.  Digital  Camera 

1 1 .  Notebook  computer  for  training 

Users 

Users  were  AC- 13  OH  Gunship  maintainers  qualified  to  use  technical  data  on  the 
Flightline.  Six  users  participated  in  the  test.  Technicians  included  Crew  Chiefs  and 
Specialists  with  varying  levels  of  expertise  from  a  3-level  to  a  7-level. 

Facilities 

The  16th  Aircraft  Maintenance  Unit  (AMU)  located  at  Hurlburt  Field  served  as  the 
test  facility.  The  test  aircraft  was  the  AC- BOH  Gunship. 

Data  Collection  Team 

The  data  collection  team  consisted  of  five  individuals:  two  on-site  coordinators, 
one  subject  matter  expert,  one  videographer,  and  an  equipment  coordinator.  The  on-site 
coordinators  arranged  for  all  subjects,  met  all  requirements  for  conducting  the  study  on 
the  flightline,  and  conducted  the  inbrief  and  outbrief  sessions.  The  subject  matter  expert 
provided  the  majority  of  the  interaction  with  each  Crew  Chief  and  Specialist.  The 
videographer  was  responsible  for  video  and  audio  documentation,  via  camcorder,  of  each 
experimental  session.  The  equipment  coordinator  was  responsible  for  ensuring  that  all 
necessary  hardware  (e.g.,  computers)  was  available  and  ready. 

Procedure 

1 .  Following  training,  each  technician  was  given  one  of  the  three  portable  devices  (full 
screen,  half  screen,  or  quarter  screen). 

2.  After  obtaining  a  flashlight  and  their  flightline  badge,  the  team  proceeded  to  the 
aircraft. 

3 .  The  participant  was  then  instructed  to  perform  the  power  on  procedure  on  the  aircraft 
with  one  of  the  three  hardware  platforms. 

4.  They  moved  around  the  aircraft  including  outside  the  aircraft,  several  locations  in  the 
cargo  area,  in  the  booth  (the  area  containing  positions  for  the  IR  Operator,  EWO,  and 
TV  Operator),  in  navigator’s  station  on  the  flight  deck,  and  the  co-pilot  and  pilot 
stations  on  the  flight  deck. 

5.  After  performing  the  procedure  they  responded  to  questions  on  the  post-condition 
questionnaire. 

6.  Participants  were  then  given  the  next  hardware  device  and  repeated  steps  three,  four 
and  five,  until  all  three  devices  had  been  used. 

7.  After  each  technician  completed  all  three  conditions,  they  filled  out  the  post-test 
questionnaire  and  discussed  problems  they  had  with  each  of  the  devices. 
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Results 


Subjective  data  were  collected  during  the  Spiral  3  field-testing  following  use  of 
each  type  of  hardware  (post-condition)  and  at  the  end  of  the  test  (post-test).  Post¬ 
condition  questions  were  built  from  issues  and  concerns  raised  by  technicians  during  the 
Spiral  3  Synthetic  test.  These  questions  dealt  with  usability  issues  such  as  readability  in 
varying  lighting,  and  physical  properties  of  the  device  such  as  ruggedization.  Post-test 
questions  were  built  using  the  same  rank  ordering  questions  posed  in  the  Spiral  1 
Usability  Test.  While  direct  comparisons  between  Spiral  1  hardware  and  Spiral  3  Field 
Test  hardware  cannot  be  made,  it  is  important  to  note  that  similar  questions  were  asked  in 
both  tests.  The  sections  below  present  results  for  the  Spiral  3  Field  Test. 

Post-Condition  Questionnaires 


Seven  questions  were  asked  following  use  of  each  type  of  hardware.  These 
questions  were  on  a  5-point  Likert  scale: 
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These  five  response  alternatives  were  selected  based  on  evidence  that  they  are  at 
least  one  standard  deviation  apart  from  one  another  and  have  parallel  wording  (Babbitt,  & 
Nystrom,  1989).  A  cluster  graph  for  each  question  is  provided  below  to  illustrate  the 
responses  provided  by  each  participant.  The  method  for  interpreting  the  cluster  graph 
was  established  a  priori  to  the  test  and  is  as  follows: 

1 .  Usability  problems  and  potential  problems  are  apparent  only  if  two  or  more 
participants  give  ratings  of  3,  4  or  5. 

2.  If  one  or  more  participants  give  a  rating  of  3,  and  only  one  participant  gives  a 
rating  of  4  or  5  then  a  potential  usability  problem  is  indicated. 

3.  If  two  or  more  participants  give  a  rating  of  4  or  5,  a  definite  usability  problem 
is  indicated. 

Using  this  set  of  guidelines.  Figure  22  indicates  that  users  had  no  difficulty 
reading  the  Toughbook  in  sunlight;  however,  they  did  seem  to  have  problems  reading  the 
LXE  device  (two  or  more  indicated  a  4  or  5).  Responses  concerning  use  of  the  Intermec 
indicate  that  there  are  potential  problems  reading  the  device  in  sunlight  conditions. 
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Ease  in  reading  the  Job  Guide  in  sunlight  conditions 


Completely  Reasonably  Borderline  Moderately  Extremely 

Acceptable  Acceptable  Unacceptable  Unacceptable 


Figure  22.  Post-Condition,  ease  in  reading  the  Job  Guide  in  sunlight  conditions 
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Figure  23  indicates  that  users  had  no  difficulty  reading  any  of  the  devices  in 
nighttime  conditions.  Nighttime  conditions  were  simulated  for  5  out  of  the  6  technicians 
by  performing  part  of  the  task  in  the  AC-130  booth  (the  area  with  stations  for  the  IR 
Operator,  EWO,  and  TV  Operator).  The  sixth  participant  performed  most  of  the  task  in 
real  nighttime  conditions. 


Ease  in  reading  the  Job  Guide  in  nighttime  conditions 
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Figure  23.  Post-Condition,  ease  in  reading  the  Job  Guide  in  nighttime  conditions 
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As  shown  in  Figure  24,  users  had  no  difficulty  reading  the  Toughbook  or 
Internee  in  the  cargo  area  (medium  lighting  conditions);  however  they  do  indicate 
potential  problems  with  reading  the  LXE  in  these  conditions. 


Ease  in  reading  the  Job  Guide  in  inside  the  aircraft  cargo  area 
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Figure  24.  Post-Condition,  ease  in  reading  the  Job  Guide  inside  the  aircraft  cargo  area 
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Figure  25  indicates  that  there  are  no  problems  manipulating  the  Job  Guide 
technical  orders  with  the  Toughbook  or  Intermec.  The  LXE,  however,  does  indicate 
some  usability  problems  manipulating  data. 


Ease  in  manipulating  data  to  make  it  viewable 
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Figure  25.  Post-Condition,  ease  in  manipulating  data  to  make  it  viewable 
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With  respect  to  carrying  the  devices,  users  indicated  no  problems  with  either  the 
LXE  or  Intermec  (see  Figure  26).  They  did  identify  potential  problems  carrying  the 
Toughbook. 


Ease  in  carrying  the  device 
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Figure  26.  Post-Condition,  ease  in  carrying  the  device 
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A  similar  pattern  of  results  is  found  with  regard  to  the  weight  of  the  devices  (see 
Figure  27).  Users  had  no  problems  with  the  weight  of  either  the  LXE  or  the  Intermec, 
but  did  indicate  potential  problems  with  the  Toughbook. 


Weight  of  the  Device 
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Figure  27.  Post-Condition,  weight  of  the  device 
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Ruggedization  was  measured  based  purely  on  perceived  ruggedization.  Users 
were  told  (if  asked)  that  all  three  systems  were  “somewhat  ruggedized.”  Therefore, 
ruggedization  was  rated  on  perceived,  rather  than  actual  ruggedization  specifications.  As 
shown  in  Figure  28,  users  perceived  potential  problems  with  the  Toughbook  and  the 
LXE,  but  did  not  indicate  similar  perceptions  with  the  Intermec. 
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Figure  28.  Post-Condition,  apparent  ruggedization  of  the  device 


Post-Test  Questionnaires 

The  Post-Test  Questionnaires  addressed  the  participants’  preference  of  equipment 
configuration  (see  Appendix  E  for  a  sample  of  questionnaires  administered).  The 
participants’  rank  ordered  the  devices  on  4  criteria:  1)  display,  2)  manipulating  data,  3) 
size  of  the  device,  4)  and  overall  technology.  The  following  data  are  averages  of  the 
participants’  rank  orders  (see  Figures  29-32). 

Cluster  graphs  are  provided  below  to  illustrate  the  rankings  provided  by  each 
participant.  Interpretation  of  the  graph  is  as  follows: 

1 .  Clusters  that  are  evenly  distributed  from  most  preferred  to  least  preferred  for 
two  or  more  hardware  devices  indicate  no  preference. 

2.  Clusters  that  are  unevenly  distributed  indicate  that  some  preference  is 
observed. 

Using  this  set  of  guidelines,  ranking  of  the  devices  for  display  clearly  shows  that 
the  Toughbook  was  the  most  preferred  display.  Users  indicated  that  the  contrast  provided 
by  this  device  was  so  good  that  they  could  read  it  from  a  distance.  The  LXE  was  ranked 
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as  the  least  preferred.  User  comments  concerning  this  display  was  that  it  was  very 
difficult  to  read;  one  user  added  that  it  was  unreadable  in  sunlight. 
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Preference  for  a  display  when  referencing  Job  Guides 


Figure  29.  Ranking  for  Display 


Preference  for  a  manipulating  data  when  referencing  Job  Guides 


Toughbook 
Full  Screen 


LXE 
Vi  Screen 


Intermec 
!4  Screen 


Toughbook  most  preferred 


LXE  least  preferred  for 
manipulating  data 


□ 


Most  Least 

Preferred  Preferred 


Figure  30.  Ranking  for  manipulating  data 
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Data  were  manipulated  on  all  three  devices  with  a  touch  screen  and  keypad.  The 
Toughbook  and  LXE  had  full  physical  QWERTY  keyboards,  while  the  Intermec  had  an 
on-screen  keyboard  with  a  physical  number  pad.  The  touch  screens  were  used  to  scroll 
around  a  given  page  (up,  down,  left  and  right).  For  the  most  part,  scrolling  on  a  page  was 
not  required  on  the  Toughbook;  a  complete  page  was  displayed  at  a  time.  For  the  LXE, 
users  were  required  to  scroll  up  and  down  on  a  given  page  (it  was  wide  enough  to  show 
the  full  width  of  the  text).  The  Intermec  required  scrolling  in  all  four  directions.  On  all 
three  devices,  moving  between  pages  was  accomplished  by  pressing  a  key  on  the  keypad 
or  touching  an  arrow  icon  on  the  screen. 

The  LXE  was  the  least  preferred  device  for  manipulating  data.  Several  users 
commented  on  the  relatively  low  sensitivity  of  the  LXE  touch  screen  (more  pressure  was 
required  to  activate  this  screen  than  the  other  two).  When  making  page-to-page 
movements,  several  users  noted  the  slow  processor  on  the  LXE.  The  Toughbook  was  the 
most  preferred  device  for  data  manipulation. 


Preference  for  a  size  of  the  device  when  referencing  Job  Guides 
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Figure  31.  Ranking  for  size  of  the  device 

As  might  be  expected,  technicians  preferred  the  size  of  the  Intermec  device  to  the 
size  of  the  other  two  devices.  The  Toughbook  was  the  least  preferred  device  for  size. 
Users  indicated  that  the  Toughbook  was  too  big  and  heavy,  and  was  awkward  to  carry. 
These  results  and  user  comments  are  consistent  with  the  Post-Condition  ratings  given  for 
the  size  and  weight  of  the  Toughbook. 
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Preference  for  a  overall  technology  when  referencing  Job  Guides 
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Figure  32.  Ranking  for  Overall  Technology 

Overall,  technicians  preferred  the  Intermec  710.  Users  indicated  that  it  was  a 
good  size  and  the  quality  of  display  was  adequate.  While  several  technicians  commented 
that  the  Toughbook  provided  the  best  display,  they  noted  that  it  was  too  heavy  to  use  on 
the  flightline. 
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Discussion 


Toughbook 

As  shown  in  the  results,  the  Toughbook  provided  users  with  a  good  quality 
display.  Participants  could  read  the  display  from  a  distance  and  were  able  to  configure 
on-screen  data  so  that  an  entire  page  of  the  Job  Guide  was  visible.  This  high  level  of 
quality  persisted  through  all  lighting  conditions.  Users  found  no  problems  manipulating 
data  with  this  device.  The  touch  screen  worked  well  to  manipulate  information  on  a 
single  page  as  well  as  between  pages. 

The  Toughbook,  however,  required  awkward  hand  and  arm  positioning  in  order  to 
carry  the  device  (see  Figure  33).  This  physical  posturing  was  due  primarily  to  the  weight 
of  the  device.  On  many  occasions,  users  were  required  to  rest  the  device  against  their 
body,  carry  it  in  their  arm  (as  opposed  to  carrying  it  with  their  hand),  or  set  it  down.  This 
weight  and  size  concern  is  reflected  in  the  preference  ratings,  in  which  technicians 
indicated  a  preference  for  the  smallest,  lightest  weight  Intermec  device. 


Figure  33.  Carrying  the  Toughbook 


LXE 

Review  of  the  information  collected  from  users  concerning  the  LXE  indicates  two 
primary  usability  problems,  the  screen  quality  and  the  sensitivity  of  the  touch  screen. 

The  screen  quality  was  a  problem  in  the  office  setting  (identified  in  Spiral  3  Synthetic 
test)  as  well  as  on  the  flightline.  The  only  lighting  environment  that  posed  no  problems 
was  night. 
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An  observation  made  by  experimenters  during  this  usability  test  was  that  the  strap 
on  the  LXE  is  placed  in  a  manner  that  promotes  ulnar  deviation  (a  contributor  to  carpal 
tunnel  syndrome).  Some  technicians  rotated  the  device  in  order  to  eliminate  the  awkward 
ulnar  deviation  promoted  by  the  strap  location  (see  Figure  34).  Simple  modification  of 
the  strap  arrangement  on  the  device  would  not  only  make  the  device  easier  to  use,  but 
would  also  promote  better  physical  health  over  the  long-term  use  of  the  device. 


Figure  34.  Ulnar  deviation  adjustment  with  the  LXE 


Intermec 

The  Intermec  was  the  most  preferred  device  overall.  Its  small  size  and  light 
weight  footprint  were  features  that  the  users  thought  were  important  features  for  this  tool. 

Users  did  indicate  potential  problems  reading  the  device  in  sunlight  conditions 
(see  Figure  35).  This  concern  is  also  reflected  in  the  ranking  of  displays  and  in  user 
comments  concerning  the  comparison  of  the  quality  of  the  displays. 
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Conclusions 


This  test  compared  three  portable  computing  devices  using  Job  Guides  for 
performing  a  Power  On  procedure  on  the  flightline.  Air  Force  Crew  Chiefs  and 
Specialists  for  the  AC- 13  OH  aircraft  participated  in  the  study.  Among  the  devices  shown, 
technicians  preferred  the  Intermec  710  computer  to  the  other  two  devices  for  viewing  the 
Power-On  procedure  on  the  flightline. 

Technicians  identified  the  Intermec  as  the  most  preferred  device  overall  for 
viewing  the  Power-On  Job  Guide  on  the  flightline.  Several  qualifications,  however,  are 
indicated  for  the  usability  of  the  device  on  the  flightline  for  Job  Guide  procedures.  Users 
identified  potential  problems  using  the  device  in  sunlight  conditions.  Along  this  line, 
users  had  a  strong  preference  for  the  Toughbook  display,  confirmed  by  specific 
comments  from  the  participants  about  the  good  quality  of  the  Toughbook  display. 

The  Toughbook  clearly  provided  the  best  display;  however  the  size  and  weight  of 
the  device  were  such  that  users  did  not  prefer  the  device  for  use  on  the  flightline  for 
procedures  such  as  the  Power  On  task. 

While  the  LXE  device  was  not  rated  or  ranked  well  in  this  usability  test,  there  are 
indications  that  improved  display  quality  and  increased  touch  screen  sensitivity  would 
make  the  LXE  at  least  a  comparable  device  to  the  Intermec. 

Finally,  while  technicians  preferred  the  Intermec  device,  it  is  important  to  note 
that  this  usability  test  did  not  include  empirical  performance  testing  of  the  accuracy  of 
maintenance  activities  when  using  any  of  these  devices.  Maintenance  accuracy  refers  to 
the  quality  of  performance  issues,  such  as  completing  all  steps  listed  in  the  Job  Guide, 
following  all  steps  as  instructed  in  the  Job  Guide,  and  following  all  safety  guidance 
provided  in  the  Job  Guide.  Empirical  testing  for  accuracy  should  be  conducted  prior  to 
implementing  Job  Guides  on  the  flightline  using  a  %  screen  device  like  the  Intermec. 


Spiral  3,  Field  Usability  Test 


54 


OVERALL  RECOMMENDATIONS 


Spiral  L  Usability  Test 

The  Spiral  1,  Usability  Test  was  the  first  in  a  series  of  spiral  tests  looking  at  the 
usability  of  computing  systems  on  the  Air  Force  maintenance  flightline. 
Recommendations  and  considerations  resulting  from  this  test  directly  affected  the  test 
design  for  the  Spiral  3  tests. 

There  are  several  recommendations  and  considerations  for  selection  of  future 
hardware  systems  that  can  be  made  using  the  results  of  this  first  test.  These 
recommendations  are  provided  below: 

•  Hardware  footprint  should  focus  on  small,  lightweight  devices.  Devices  that 
can  be  held  with  one  hand  are  preferable.  The  size  and  weight  of  the  Libretto  and 
FM-Net  devices  were  desirable  to  technicians  in  the  first  spiral.  Note  that  while 
the  Libretto  was  a  full  computer  with  a  full  screen,  technicians  frequently  used  a 
one-hand  carry  for  the  device. 

•  If  any  manual  text  entry  (numbers  and  letters)  is  required,  the  device  should 
offer  a  full,  physical  keyboard.  In  Spiral  1,  technicians  had  some  difficulty  with 
the  on-screen  keyboard — especially  with  respect  to  screen  real  estate.  Note, 
however,  that  physical  key  size  does  not  necessarily  have  to  be  standard  size.  The 
smaller  size  of  the  Libretto  keys  did  not  pose  any  noted  difficulties. 

•  Voice  recognition  should  be  offered  wherever  practical,  and  especially  for 
field  selection.  Text  entry  is  also  still  a  prime  area  for  entry  of  data  by  voice; 
however,  vocabularies  need  to  be  further  modified  to  allow  wider  acceptance  of 
maintenance  terminology. 

•  Information  entered  verbally  (e.g.,  via  radio)  should  allow  technicians  to 
confirm  the  information  uttered.  This  confirmation  could  be  through  a  display; 
however,  it  could  also  be  provided  to  the  technician  through  other  means,  such  as 
verbal  feedback  of  information  entered. 

•  The  computer  should  include  as  much  ruggedization  as  is  possible  without 
compromising  weight  and  size  of  the  device.  Many  manufacturers  currently  use 
a  4-foot  drop  test.  At  a  minimum,  this  criterion  should  be  included  where 
possible. 

•  A  touch  screen  should  be  included  for  field  selection.  The  device  should  offer 
a  tethered  pointing  device.  Technicians  indicated  the  desire  for  this  feature  on  the 
two  devices  in  Spiral  1  that  offered  touch  screen  capability  (Itronix  and  Xybemaut 
Wrist). 

In  addition  to  these  recommendations,  there  was  at  least  one  area  where  data  from 
the  Spiral  1  test  was  inconclusive.  This  area  should  be  further  investigated  in  subsequent 
tests. 

•  The  first  spiral  tested  a  full  640  x  480  VGA  screen  on  the  Itronix,  Libretto, 
Xybemaut  HMD,  and  the  Xybemaut  Wrist  devices.  Smaller  screen  sizes,  such  as 
U  or  14  screen  VGA  were  not  compared.  Future  tests  should  look  specifically  at 


55 


determining  the  usability  of  these  small  screens  for  viewing  Technical  Manuals 
and  forms  completion  on  the  flightline. 

Spiral  3,  Synthetic  Usability  Test 

The  purpose  of  the  Spiral  3  Synthetic  test  was  to  evaluate  and  determine  the 
usability  of  technical  data  given  the  different  screen  sizes  of  the  Panasonic  Toughbook 
(full  screen),  LXE  (half  screen),  and  Intermec  (quarter  screen).  The  study  was  designed 
to  control  environmental  factors,  such  as  sunlight  and  temperature.  Based  upon  the 
findings  of  the  Synthetic  test,  several  recommendations  and  considerations  can  be  made 
for  future  testing.  These  recommendations  and  considerations  are  provided  below: 

•  Smaller  technical  data  formats,  such  as  Job  Guides  and  work  cards,  may  be 
suitable  for  quarter  screen  devices  like  the  Intermec  710.  However,  these 
devices  should  be  tested  in  a  flightline  environment  to  determine  whether  they  are 
usable  and  useful  for  these  display  purposes. 

•  Larger  data  formats,  such  as  fault  reporting  manuals,  fault  isolation  trees, 
and  schematics,  may  only  be  suitable  for  full  screen  computers  like  the 
Panasonic  Toughbook.  The  smaller  devices,  such  as  the  LXE  and  Intermec,  do 
not  allow  the  viewing  space  necessary  to  maneuver  around  the  large  images 
inherent  in  these  formats.  Full  screen  devices  should  also  be  tested  in  a  flightline 
environment  to  determine  whether  they  are  usable  and  useful  for  these  large  data 
display  purposes. 

Spiral  3.  Field  Usability  Test 

The  Spiral  3  Field  test  took  place  in  a  flightline  environment  and  evaluated  the 
usability  of  Power-On  Job  Guides  on  three  portable  maintenance  aids.  For  this  task, 
maintainers  found  the  Intermec  710  to  be  the  preferred  device.  While  findings  indicate 
that  a  quarter  screen  device  might  be  suitable  for  displaying  Job  Guides,  there  are  several 
recommendations  and  considerations  that  need  to  be  addressed.  These  recommendations 
and  considerations  are  provided  below: 

•  Future  tests  should  evaluate  the  user’s  accuracy  in  using  the  technical  data 
displayed  on  the  electronic  aid.  For  example,  did  the  technician  miss  steps,  or 
perform  steps  incorrectly  when  using  the  device. 

•  The  Spiral  3,  Field  test  was  performed  using  the  Power-On  maintenance 
task.  Future  tests  should  include  different  Job  Guide  procedures  in  order  to 
gain  a  better  understanding  of  the  effectiveness  of  the  quarter  screen  device 
on  various  maintenance  tasks. 

•  Any  device  to  be  used  on  the  flightline  should  be  tested  in  sunlight  conditions 
on  the  flightline  to  assure  good  display  quality.  This  is  true  for  any  screen  size. 

•  Hand-held  devices,  such  as  the  LXE-MXE  and  the  Intermec  710  are,  by 
design,  limited  in  their  memory.  Everything,  including  memory,  is  designed  to 
minimize  drain  on  the  battery.  These  designs,  however,  also  limit  the  amount  of 
information  that  can  be  stored  on  the  device.  Before  these  small  devices  can  be 
used  to  display  technical  data  on  the  flightline,  memory  limitations  associated 
with  them  needs  to  be  addressed.  Most  current  formats  for  technical  data  require 
more  memory  than  is  available  on  these  devices.  An  increased  memory  capacity 
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would  allow  complete  and/or  multiple  Job  Guides  to  be  stored  and  presented  on 
these  small  devices. 
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APPENDIX  A  -  HARDWARE  SPECIFICATIONS 


Spiral  1  Hardware 


Company 

Itronix 

Xybernaut 

Toshiba 

Product 

X-C  6250  Pro 

MA  IV 

Libretto  70CT,  11  OCT 

Product  Type 

Portable  Computer 

Wearable  Computer 

Portable/Handheld  Computer 

Platform 

MS  Windows  95/98 

Windows  95,98,NT 

Windows  95,  98,  NT 

2000,  ME 

CPU 

300  MHz  GXm 

200/233  Pentium  MMX 

Intel  Pentium  90MHz  or  120MHz 

Memory 

256  MB 

32/64/1 60MB 

32MB 

Storage 

6GB  or  12GB 

2.1GB  or  4.3GB 

1.19GB  or  3.95GB 

Dimension  (H  x  W  x  D) 

10.5  x  7.5  x  3.0  in. 

2.5  x  7.5  x  4.6  in. 

Weight 

6.9  pounds 

1.75  pounds 

Expansion  Slots 

2  CardBus  slots 

PC  Card  Type 

2  Type  1  or  II 

1  Type  III 

Audio 

Soundblaster  compatible 

Audio  Speaker,  external  1 

audio  chip  (ESS  Tech) 

speaker  jack 

full  duplex 

Display 

SVGA  color 

FPD:  640x480  VGA 

640x480  256  color 

Colorvue  sunlight  viewable 

touchscreen 

optional  touchscreen 

sunlight  readable 

HMD  (display  unit):  640x480 

Display  Dimension 

10.4" 

FPD:(L)  7.5  in,  (W)  4.7  in, 

(D)  1.6  in 

HMD:(L)  2.4  in,  (W)  3.5  in, 

(D)  10.1  in 

Display  Weight 

FPD:  520  g 

N/A 

HMD:  280  g 

Ruggedization  i 

54  repeated  drops,  vibration 
Df.04g2/Hz  over  20-1 000  Hz 
'andom 

No 

Battery  Life  ; 

2/3  or  5/6  hours  ( 

3  hours 

RF  Comm  and  ! 

56  Kbps 

PCMCIA  Radio  Card 

Other  Communications  ( 

3DPD/Cellular  networks 
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Spiral  3  Hardware 


Company 

Intermec 

LXE 

Panasonic 

Product 

700  Series  Mobile  Comp. 

MX3 

Toughbook  34 

Product  Type 

Handheld 

Handheld 

Portable  Computer 

Platform 

MS  Pocket  PC 

488  PC  platform 

Windows  98/95/NT/2000 

MS-DOS,  DOS  compatible 

CPU 

Intel  Pentium  III  400MHz 

SA-1 1 1 0  Rise  Processor 

Memory 

Storage 

32MB,  (32,64,96,128MB 
cpf) 

10GB-12GB 

Dimension  (H  x  W  x  D) 

(H)  3.5  in,  (W)  1.5  in.,  (L) 
7.5in. 

Weight 

16  ounces  (11b) 

3.8lbs 

Expansion  Slots 

Type  llxl 

Z V  (Zoomed  Video)  Port  and  Card  Bus 

Audio 

No 

Crystal  CS4229  AC-97  v2.1  Compliant 
Audio  Codec 

Integrated  Speaker 

Microphone 

Display 

Monochrome  LCD 
240x320 

640x280  VGA  LCD 

8.4in  800x600  TFT  Active  Matrix  Color 

LCD  with 

Touchscreen 

DayBrite  ARX  anti-reflective  LCD  for 
viewing  in  sunlight 

Silicon  Motion  video  controller,  4MB 
VRAM 

Display  Dimension 

Display  Weight 

N/A 

N/A 

Ruggedization 

4ft  drop 

4ft  drop 

yes 

Battery  Life 

8-10  hours 

7  hours 

RF  Comm  and 

900MHz  or  2.4GHz  radio 

nteqrated  56Kbps 

Other  Communications 

1 0  Base-T  Ethernet 

l_AN:  802.11b  (Wi-Fi 
certified) 

WAN:  GSM,  BellSouth, 
CDPD, 

Motient 

PAN:  Bluetooth 

Appendix  A 


60 


APPENDIX  B  -  SPIRAL  1,  USABILITY  TEST  QUESTIONNAIRES 

PRE-TEST  QUESTIONNAIRE 

Date: _ _ 

All  information  will  remain  confidential. 

Name _  Rank _ 

AFSC _  Time  in  Air  Force _ Time  in  AFSOC _ 

Time  on  C-130s _  Other  Weapon  Systems  worked _ 

Current  Job  Title _  Years  of  Experience  at  Current  Job 

Previous  Jobs  with  Years  of  Experience 


Circle  the  response  that  best  applies  to  you. 

1 .  Approximately  how  often  do  you  open  work  orders  on  the  C-130? 

Daily  Weekly  Monthly  Rarely  (less  than  once  a  month) 

2.  Approximately  how  often  do  you  use  a  personal  computer  (on  the  job  or  at  home)? 

Daily  Weekly  Monthly  Rarely  (less  than  once  a  month) 

3.  Approximately  how  often  do  you  use  open  work  orders  on  the  CAMS  terminal? 

Daily  Weekly  Monthly  Rarely  (less  than  once  a  month) 

4.  Have  you  previously  participated  in  an  AFRL  field  evaluation? 

Yes  No 

5.  If  you  answered  yes  to  the  previous  question,  when  and  what  did  you  evaluate? 
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[Insert  device  name  here] 

POST-CONDITION  QUESTIONNAIRE 


Date: _ 

Subject  #: _ 

All  information  will  remain  confidential. 

[insert  device  name  here]  at  the  aircraft: 

1.  Using  the  [insert  device  name  here]  to  open  work  orders  was: 

Easy  Not  at  all  easy 


1  2  3  4  5 


2.  If  you  could  change  anything  on  the  [insert  device  name  here],  what  would  it  be? 


3.  What  features  did  you  like  about  [insert  device  name  here]? 


4.  What  features  did  you  NOT  like  about  [insert  device  name  here]? 


5.  What  other  comments  do  you  have? 
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POST-TEST  QUESTIONNAIRE 


All  information  will  remain  confidential. 


Date: 
Subject  #: 


1 .  Rank  order  your  preference  for  a  display  device  for  opening  a  work  order  on  the  flightline 
(1  is  first  choice,  4  is  last  choice): 

Choices  are:  a)  Itronix,  b)  Libretto,  c)  head-mounted,  d)  wrist- worn 

1.  _ 

2.  _ 

3.  _ 

4. 


Comments: 


2.  Rank  order  your  preference  for  entering  text  when  opening  a  work  order  on  the  flightline  (1  is  first 
choice,  5  is  last  choice): 

Choices  are:  a)  voice,  b)  Itronix  full  keyboard,  c)  Libretto  small  keyboard,  d)  wrist- worn  keyboard, 
e)  wrist-worn  on-screen  keyboard 

1.  _ 

2.  _ 

3.  _ 

4.  _ 

5.  "  . 


Comments: 
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3.  Rank  order  your  preference  for  selecting  fields  when  opening  a  work  order  on  the  flightline  (1  is  first 
choice,  5  is  last  choice): 

Choices  are:  a)  voice,  b)  Itronix,  TrackPoint  mouse,  c)  Libretto  thumb  pointer,  d)  hip-mounted 
pointing  device,  e)  touch  screen 

1.  _ 

2.  _ 

3.  _ 

4.  _ 

5.  _ 


Comments: 


4.  Overall,  rank  order  your  preference  for  one  overall  technology  for  opening  a  work  order  on  the 
flightline  (1  is  first  choice,  5  is  last  choice): 

Choices  are:  a)  FM  radio,  b)  Itronix  computer,  c)  Libretto  computer,  d)  Xybemaut  wearable  computer 
with  head-mounted  display,  e)  Xybemaut  wearable  computer  with  wrist-worn  display, 

L  _ 

2.  _ 

3.  _ _ 

4.  _ 

5. 


Comments: 


4.  What  other  comments  do  you  have? 
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APPENDIX  C  -  SPIRAL  3  SYNTHETIC  USABILITY  TEST  FOR  TECHNICAL 

DATA 

PRE-TEST  QUESTIONNAIRE 

SUBJECT  NUMBER: _ 

DATE: _ 

EXPERIMENTER:  JJ  CD  LQ 

All  information  will  remain  confidential. 

Name _  Rank _ 

AFSC _  Time  in  Air  Force _ Time  in  AFSOC _ 

Time  on  C-130s _  Other  Weapon  Systems  worked _ 

Current  Job  Title _ Years  of  Experience  at  Current  Job _ 

Previous  Jobs  with  Years  of  Experience  (e.g.,  other  AFSCs) 


Circle  the  response  that  best  applies  to  you. 

6.  Approximately  how  often  do  you  use  technical  data  on  the  C-130? 

Daily  Weekly  Monthly  Rarely  (less  than  once  a  month) 

7.  Approximately  how  often  do  you  use  a  personal  computer  (on  the  job  or  at  home)? 

Daily  Weekly  Monthly  Rarely  (less  than  once  a  month) 

8.  Have  you  previously  participated  in  an  AFRL  field  evaluation? 

Yes  No 

9.  If  you  answered  yes  to  the  previous  question,  when  and  what  did  you  evaluate? 
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POST-CONDITION  QUESTIONNAIRE 

SUBJECT  NUMBER: _ 

DATE: _ 

EXPERIMENTER:  JJ  CD  LQ 

Device:  full  half  quarter 

All  information  will  remain  confidential.  Circle  or  underline  your  response. 

Data  Format  1 : 

Starting  at  the  Table  of  Contents,  find  the  Action  Taken  code  for  “BENCH 
CHECKED-TRANSFERRED  TO  ANOTHER  BASE  OR  UNIT” 

1 .  Ease  in  the  finding  information  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

2.  Ease  of  reading  information  in  this  format  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

Data  Format  2: 

Review  the  Input  Conditions  for  ’’CONNECTING  AND  DISCONNECTING 
EXTERNAL  POWER  (MC-130E  AIRPLANES).” 

3.  Ease  of  reading  information  in  this  format  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

Find  the  step  lc  and  the  associated  graphic  for  ”  CONNECTING  AND 
DISCONNECTING  EXTERNAL  POWER  (MC-130E  AIRPLANES).” 

4.  Ease  in  finding  the  associated  graphic  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 
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5.  Ability  to  associate  the  numerical  reference  points  between  the  graphic  and  text  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

Data  Format  3: 

Find  the  page  number  for  ”  FLIGHT  STATION  LIGHTING.” 

6.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.,  scrolling)  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

Go  to  Page  8.  Read  Section  2.3  until  you  get  to  the  sentence  starting  at  “The  utility 
lights  have  a  momentary  switch  ...” 

7.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.,  scrolling)  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

Go  to  Page  5.  Find  Section  2.2  “FLIGHT  STATION  DOME  AND  THUN¬ 
DERSTORM  LIGHTS”  and  find  the  first  reference  to  a  graphic,  and  find  that 
graphic 

8.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.,  scrolling)  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

Data  Format  4: 


Go  to  Page  3  and  read  the  first  step. 


9.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.,  scrolling)  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 
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Scroll  to  the  next  card  and  view  the  “Tail  Cone.” 


1 0.  Ease  in  finding  the  information  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

1 1 .  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.,  scrolling)  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

4 

5 

Data  Format  5: 


Go  to  Page  10,  in  Test  5  assume  that  the  clear  light  comes  on  and  go  to  the  next  Test. 


12.  Ease  in  finding  the  information  was: 


Completely 

acceptable 

1 

Reasonably 

acceptable 

2 

Borderline 

3 

Moderately 

unacceptable 

4 

Extremely 

Unacceptable 

5 

Go  to  Test  11;  find  the  graphic  illustrating  the  “ 

‘Thunderstorm  Light  Switch” 

13.  Ease  in  finding 

the  information  was: 

Completely 

acceptable 

1 

Reasonably 

acceptable 

2 

Borderline 

3 

Moderately 

unacceptable 

4 

Extremely 

Unacceptable 

5 

Data  Format  6: 


Go  to  Page  17;  go  to  Fault  Report  “Does  Airplane  Maintain  Desired  Cabin 
Altitude?”  Find  the  Fault  Code  associated  with  the  response  “No”  and  read  the 
AFTO  781A  Report  description. 


14.  Ease  in  finding  the  information  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 
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15.  Ease  in  maintaining  your  orientation  (e.g 

,  situation  awareness  or  “You  are  Here” 

orientation)  was: 

Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline  unacceptable 

Unacceptable 

1 

2 

3  4 

5 

16.  Ease  in  manipulate 

g  data  so  that  it  was  easily  viewable  (e.g.,  scrolling)  was: 

Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline  unacceptable 

Unacceptable 

1 

2 

3  4 

5 

Data  Format  7: 

Read  the  Notes  on  Page  2. 


1 7.  Ease  in  reading  information  in  this  format  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

o 

J 

4 

5 

Data  Format  8: 


Go  to  page  5;  trace  the  NVIS  Master  Relay  #3  to  the  Thunderstorm  Lights. 

18.  Ease  in  reading  information  in  this  format  was: 


Completely 

acceptable 

1 

Reasonably 

acceptable 

2 

Borderline 

3 

Moderately 

unacceptable 

4 

Extremely 

Unacceptable 

5 

19.  Ease  in  maintaining  your  orientation  (e 

.g.,  situation  awareness  or  “You  are  Here” 

orientation)  was: 

Completely 

acceptable 

1 

Reasonably 

acceptable 

2 

Borderline 

o 

J 

Moderately 

unacceptable 

4 

Extremely 

Unacceptable 

5 
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Data  Format  9: 


Go  to  Page  10;  then  page  to  Page  14.  Follow  the  circuit  from  28V  circuit  breaker 
and  follow  the  circuit  to  the  ground  switches. 


20.  Ease  in  finding  the  information  was: 


Completely 

acceptable 

1 

Reasonably 

acceptable 

2 

Borderline 

3 

Moderately 

unacceptable 

4 

Extremely 

Unacceptable 

5 

2 1 .  Ease  in  manipulating  data  for  tracing  signals  (e.g.,  scrolling)  was: 

Completely 

acceptable 

1 

Reasonably 

acceptable 

2 

Borderline 

3 

Moderately 

unacceptable 

4 

Extremely 

Unacceptable 

5 

Data  Format  10: 

Go  to  Page  7  and  find  the  name  for  Figure  2-16. 

22.  Ease  in  reading 

information  in  this  format  was: 

Completely 

acceptable 

1 

Reasonably 

acceptable 

2 

Borderline 

3 

Moderately 

unacceptable 

4 

Extremely 

Unacceptable 

5 

Find  terminal  4  on  the  “Pilot  Instrument  Panel”, 
the  associated  Sheet. 

and  then  find  the  entry  point  on 

23.  Ease  in  finding  the  information  was: 

Completely 

acceptable 

1 

Reasonably 

acceptable 

2 

Borderline 

3 

Moderately 

unacceptable 

4 

Extremely 

Unacceptable 

5 

24.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.,  zooming)  was: 

Completely 

acceptable 

1 

Reasonably 

acceptable 

2 

Borderline 

o 

Moderately 

unacceptable 

4 

Extremely 

Unacceptable 

5 
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APPENDIX  D:  ANOVA  SUMMARY  TABLES 


Data  Format  1 

Starting  at  the  TOC,  find  the  Action  Taken  code  for  "Bench  Checked-Transferred 
to  another  base  or  unit" 


1  .Ease  in  finding  in 

formation 

ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

F  crit 

Between 

6.711111 

2 

3.355556 

5.311558 

0.009 

3.219938 

Within 

26.53333 

42 

0.631746 

Total 

33.24444 

44 

2.  Ease  of  reading  information  in  this  format  was 


ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

F  crit 

Between 

16.31111 

2 

8.155556 

10.93191 

0.000 

3.219938 

Within 

31.33333 

42 

0.746032 

Total 

47.64444 

44 

Data  Format  2 

Review  the  Input  Conditions  for  ’’CONNECTING  AND  DISCONNECTING 
EXTERNAL  POWER  (MC-130E  AIRPLANES).” 

3  .Ease  of  reading  information  in  this  format  was 


ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

F  crit 

Between 

10 

2 

5 

6.402439 

0.004 

3.219938 

Within 

32.8 

42 

0.780952 

Total 

42.8 

44 

Find  the  step  lc  and  the  associated  graphic  for  ”  CONNECTING  AND 
DISCONNECTING  EXTERNAL  POWER  (MC-130E  AIRPLANES).” 


4.  Ease  in  finding  the  associated 


graphic  was 


ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

F  crit 

Between 

4.577778 

2 

2.288889 

3.897297 

0.028 

3.219938 

Within 

24.66667 

42 

0.587302 

Total 

29.24444 

44 
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5.  Ability  to  associate  the  numerical  reference  points  between  the  graphic  and  text  was 


ANOVA 

Source 

SS 

df 

MS  F  P-value  F  crit 

9.244444 

2 

4.622222  6.618182  0.003  3.219938 

E 

29.33333 

42 

0.698413 

Total 

38.57778 

44 

Data  Format  3 

Find  the  page  number  for  ”  FLIGHT  STATION  LIGHTING.” 

6.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.  scrolling)  was 


ANOVA 

Source  j 

SS 

df 

MS  F  P-value  F  crit 

8.844444 

2 

4.422222  6.697115  0.003  3.219938 

27.73333 

42 

0.660317 

Total 

36.57778 

44 

Go  to  Page  8.  Read  Section  2.3  until  you  get  to  the  sentence  starting  at  “The  utility 
lights  have  a  momentary  switch  ...” 

7.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.  scrolling)  was 


ANOVA 

Source 

SS 

df 

MS  F  P-value  F  crit 

9.644444 

2 

4.822222  5.149153  0.010  3.219938 

39.33333 

42 

0.936508 

Total 

48.97778 

44 

Go  to  Page  5.  Find  Section  2.2  “FLIGHT  STATION  DOME  AND  THUN¬ 
DERSTORM  LIGHTS”  and  find  the  first  reference  to  a  graphic,  and  find  that 
graphic 

8.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.,  scrolling)  was 


ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

F  crit 

9.644444 

2 

4.822222 

6.329167 

0.004 

3.219938 

Within 

32 

42 

0.761905 

Total 

41.64444 

44 
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Data  Format  4 

Go  to  Page  3  and  read  the  first  step. 

9.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.,  scrolling)  was 


ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

F  crit 

Between 

10.53333 

2 

5.266667 

5.184375 

0.010 

3.219938 

Within 

42.66667 

42 

1.015873 

Total 

53.2 

44 

Scroll  to  the  next  card  and  view  the  “Tail  Cone.” 


ANOVA 

Source 

SS 

df 

MS 

F 

P-vaiue 

F  crit 

Between 

4.444444 

2 

2.222222 

3.482587 

0.040 

3.219938 

Within 

26.8 

42 

0.638095 

Total 

31.24444 

44 

1 1 .  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g  scrolling)  was 


ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

F  crit 

Between 

8.044444 

2 

4.022222 

6.961538 

0.002 

3.219938 

Within 

24.26667 

42 

0.577778 

Total 

32.31111 

44 

Data  Format  5 

Go  to  Page  10,  in  Test  5  assume  that  the  clear  light  comes  on  and  go  to  the  next  Test. 


12.  Ease  in 


inding  the  information  was 


ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

F  crit 

Between 

'■)  1  <*>  o  <■>  ^  o 
Z.  1 J 

2 

1.066667 

1.787234 

0.180 

3.219938 

Within 

25.06667 

42 

0.596825 

Total 

27.2 

44 

Go  to  Test  11;  find  the  graphic  illustrating  the  “Thunderstorm  Light  Switch’ 


ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

F  crit 

Between 

1.377778 

2 

0.688889 

0.844358 

0.437 

3.219938 

Within 

34.26667 

42 

0.815873 

Total 

35.64444 

44 
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Data  Format  6 

Go  to  Page  17;  go  to  Fault  Report  “Does  Airplane  Maintain  Desired  Cabin 
Altitude?”  Find  the  Fault  Code  associated  with  the  response  “No”  and  read  the 
AFTO  781A  Report  description. 

14.  Ease  in  finding  the  information  was  _ _ _ 


ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

F  crit 

Between 

10.53333 

2 

5.266667 

3.796339 

0.031 

3.219938 

Within 

58.26667 

42 

1.387302 

Total 

68.8 

44 

15.  Ease  in  maintaining  your  orientation  (e.g.,  situation  awareness  or  “You  are  Here” 

orientation)  was  _  _ _ _ _ 

ANOVA  I  1  1111 


Source 

SS 

df 

MS 

F 

P-value 

F  crit 

Between 

10.53333 

2 

5.266667 

4.365789 

0.019 

3.219938 

Within 

50.66667 

42 

1.206349 

Total 

1  61.2 

44 

16.  Ease  in  manipulating  data  so  that  it  was  easily  viewable  (e  g,,  scrolling)  was 

ANOVA  I  1 _ 

Source _ SS _ df _ MS _ F _ P-value  F  crit 

Between  5.911111  2  2.955556  3.0625  0.057  3.219938 

Within  40.53333  I  42  0.965079  I  I  I 


Data  Format  7 


Read  the  Notes  on  Page  2. 
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Data  Format  8 

Go  to  page  5;  trace  the  NVIS  Master  Relay  #3  to  the  Thunderstorm  Lights. 


18.  Ease  in  reading  ini 

formation  in  this  format  was 

ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

F  crit 

Between 

14.53333 

2 

7.266667 

6.271233 

0.004 

3.219938 

Within 

48.66667 

42 

1.15873 

Total 

63.2 

44 

19.  Ease  in  maintaining  your  orientation  (e.g.,  situation  awareness  or  “You  are  Here” 
orientation)  was _ 


ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

F  crit 

Between 

13.91111 

2 

6.955556 

6.482249 

0.004 

3.219938 

Within 

45.06667 

42 

1.073016 

Total 

58.97778 

44 

Data  Format  9 

Go  to  Page  10;  then  page  to  Page  14.  Follow  the  circuit  from  28V  circuit  breaker 
and  follow  the  circuit  to  the  ground  switches. 


20.  Ease  in  ] 

inding  the  information  was 

ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

F  crit 

Between 

15.24444 

2 

7.622222 

5.185745 

0.010 

3.219938 

Within 

61.73333 

42 

1.469841 

Total 

76.97778 

44 

21 .  Ease  in  manipulating  data  for  tracing  signals  (e.g.,  scrolling)  was 


ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

Fcrit 

Between 

16.93333 

2 

8.466667 

5.56785 

0.007 

3.219938 

Within 

63.86667 

42 

1.520635 

Total 

80.8 

44 

Data  Format  10 

Go  to  Page  7  and  find  the  name  for  Figure  2-16. 


22.  Ease  in  reading  ini 

formation  in  this  format  was 

ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

Fcrit 

Between 

24.31111 

2 

12.15556 

13.675 

0.000 

3.219938 

Within 

37.33333 

42 

0.888889 

Total 

61.64444 

44 
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Find  terminal  4  on  the  “Pilot  Instrument  Panel”,  and  then  find  the  entry  point  on 
the  associated  Sheet. 


23.  Ease  in 


inding  the  information  was 


mm 

ss 

df 

F 

P-value 

F  crit 

17.73333 

2 

0.000 

3.219938 

SSSSS9HH 

37.46667 

42 

Total 

55.2 

44 

24.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g  zooming)  was 


ANOVA 

Source 

SS 

df 

MS 

F 

P-value 

F  crit 

Between 

12.31111 

2 

6.155556 

6.686207 

0.003 

3.219938 

38.66667 

42 

0.920635 

Total 

50.97778 

44 
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Means  and  Standard  Deviations 


Data  Format  1 : 

Starting  at  the  TOC,  find  the  Action  Taken  code  for  "Bench  Checked-Transferred  to  another  base 
or  unit" 

1.  Ease  in  finding  the  information 

Mean  SD 

Toughbook  1.20  0.171 

LXE  2.13  1.267 

Intermec  1.80  0.457 


2.  Ease  of  reading  information  in  this  format  was 

Mean  SD 

Toughbook  1.13  0  124 

LXE  2.60  1.400 

intermec  2.00  0.714 

Data  Format  2: 

Review  the  Input  Conditions  for  "CONNECTING  AND  DISCONNECTING  EXTERNAL  POWER  (MC- 
130E  AIRPLANES).” 

3.  Ease  of  reading  information  in  this  format  was 

Mean  SD 

Toughbook  1.07  0.067 

LXE  2.07  1.352 

Intermec  2.07  0.924 

Find  the  step  1c  and  the  associated  graphic  for  ”  CONNECTING  AND  DISCONNECTING 
EXTERNAL  POWER  (MC-130E  AIRPLANES).” 

4.  Ease  in  finding  the  associated  graphic  was 

Mean  SD 

Toughbook  1.27  0.210 

LXE  2.00  0.857 

Intermec  1.87  0.695 


5.  Ability  to  associate  the  numerical  reference  points  between  the  graphic  and  text  was 

Mean  SD 

Toughbook  1.20  0.171 

LXE  2.27  1.210 

Intermec  2.00  0.714 

Data  Format  3: 

Find  the  page  number  for  ”  FLIGHT  STATION  LIGHTING.” 

6.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.,  scrolling)  was 

Mean  SD 

Toughbook  1.20  0.171 

LXE  2.20  1.171 

Intermec  2.07  0.638 
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Go  to  Page  8.  Read  Section  2.3  until  you  get  to  the  sentence  starting  at  “The  utility  lights  have  a 
momentary  switch  ...” 

7.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.,  scrolling)  was 


Mean 

SD 

Toughbook 

1.33 

0.381 

LXE 

2.20 

1.314 

Intermec 

2.40 

1.114 

Go  to  Page  5.  Find  Section  2.2  “FLIGHT  STATION  DOME  AND  THUN-DERSTORM  LIGHTS” 

and 

find  the  first  reference  to  a  graphic,  and  find  that  graphic 

8.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.,  scrolling)  was 

Mean, 

SD 

Toughbook 

1.13 

0.124 

LXE 

1.67 

0.667 

Intermec 

2.27 

1.495 

Data  Format  4: 

Go  to  Page  3  and  read  the  first  step. 

9.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.,  scrolling)  was 

Toughbook 
LXE 

Intermec 

Scroll  to  the  next  card  and  view  the  “Tail  Coiie.” 

10.  Ease  in  finding  the  information  was 

Toughbook 
LXE 

Intermec 


11.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.,  scrolling)  was 

Mean  SD 


Toughbook 

1.07 

0.067 

LXE 

1.80 

0.457 

Intermec 

2.07 

1.210 

Mean  SD 

1.27  0.210 

1.93  0.638 

1.93  1.067 


Mean  SD 

1.13  0.124 

2.00  1.143 

2.27  1.781 


Data  Format 5:  -  -  •  ’  \  ; 

Goto  Page  10,  in  Test  5  assume  that  the  clear  light  comes  on  and  go  to  the  next  Test. 

12.  Ease  in  finding  the  information  was 

Mean 


Toughbook 

1.60 

LXE 

1.87 

Intermec 

2.13 

0.400 

0.410 

0.981 
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Go  to  Test  11;  find  the  graphic  illustrating  the  “Thunderstorm  Light  Switch” 

13.  Ease  in  finding  the  information  was 


Mean 

SD 

Toughbook 

1.67 

0.381 

LXE 

2.00 

0.714 

Intermec 

2.07 

1.352 

Data  Format  6: 


Go  to  Page  17;  go  to  Fault  Report  “Does  Airplane  Maintain  Desired  Cabin  Altitude?”  Find  the 
Fault  Code  associated  with  the  response  “No”  and  read  the  AFTO  781A  Report  description. 

14.  Ease  in  finding  the  information  was 

Mean  SD 


Toughbook 

LXE 

Intermec 


1.40  0.400 

2.27  1.924 

2.53  1.838 


15.  Ease  in  maintaining  your  orientation  (e.g.,  situation  awareness  or  “You  are  Here”  orientation)  was 


Toughbook 

LXE 

Intermec 


Mean  SD 

1.47  0.410 

2.60  1.829 

2.33  1.381 


16.  Ease  in  manipulating  data  so  that  it  was  easily  viewable  (e.g.,  scrolling)  was 

Toughbook 

LXE 

Intermec 


Mean  SD 

1.40  0.400 

2.00  1.143 

2.27  1.352 


Data  Format  7: 

Read  the  Notes  on  Page  2. 

17.  Ease  in  reading  information  in  this  format  was 


Mean 

SD 

Toughbook 

1.07 

0.071 

LXE 

2.07 

1.352 

Intermec 

2.13 

1.695 

Data  Format  8: 

Go  to  page  5;  trace  the  NVIS  Master  Relay  #3  to  the  Thunderstorm  Lights. 

18.  Ease  in  reading  information  in  this  format  was 

Mean 

SD 

Toughbook 

1.33 

0.238 

LXE 

2.47 

1.410 

Intermec 

2.60 

1.829 

19.  Ease  in  maintaining  your  orientation  (e.g.,  situation  awareness  or  “You  are  Here”  orientation)  was 

Mean  SD 

Toughbook 

1.20 

0.171 

LXE 

2.27 

0.781 

Intermec 

2.47 

2.267 
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Data  Format  9: 

Go  to  Page  10;  then  page  to  Page  14.  Follow  the  circuit  from  28V  circuit  breaker  and  follow  the 
circuit  to  the  ground  switches. 

20.  Ease  in  finding  the  information  was 

Mean  SD 

Toughbook  1.60  0.686 

LXE  2.80  1.314 

Intermec  2.87  2.410 

21.  Ease  in  manipulating  data  fortracing  signals  (e.g.,  scrolling)  was 

Mean  SD 

Toughbook  1.53  0.552 

LXE  2.80  1.886 

Intermec  2.87  2.124 

Data  Format  10: 

Go  to  Page  7  and  find  the  name  for  Figure  2-16. 

22.  Ease  in  reading  information  in  this  format  was 

Mean  SD 

Toughbook  1.20  0.171 

LXE  2.07  1.067 

Intermec  3.00  1.429 

Find  terminal  4  on  the  “Pilot  Instrument  Panel”,  and  then  find  the  entry  point  on  the  associated 
:Sheet.  _  •  :•  " 

23.  Ease  in  finding  the  information  was 

.  Mean  SD 

Toughbook  1.40  0.400 

LXE  2.27  0.924 

Intermec  2.93  1.352 


24.  Ease  in  manipulating  data  so  that  it  is  easily  viewable  (e.g.,  zooming)  was 

Mean  SD 

Toughbook  1.33  0.238 

LXE  2.13  0.838 

Intermec  2.60  1 .686 
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APPENDIX  E  -  SPIRAL  3,  FIELD  USABILITY  TEST  QUESTIONNAIRES 

POST-CONDITION  QUESTIONNAIRE 

SUBJECT  NUMBER: _ 

DATE: _ 

EXPERIMENTER:  JJ  CD  LQ 

Device:  full  half  quarter 

All  information  will  remain  confidential.  Circle  or  underline  your  response. 

25.  Ease  in  reading  the  Job  Guide  in  sunlight  conditions  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

26.  Ease  in  reading  the  Job  Guide  in  nighttime  conditions  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

27.  Ease  in  reading  the  Job  Guide  inside  the  aircraft  cargo  area  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

28.  Ease  manipulating  data  to  make  it  viewable  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

29.  Ease  in  carrying  the  device  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

30.  The  weight  of  the  device  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 

31.  Apparent  ruggedization  of  the  device  was: 


Completely 

Reasonably 

Moderately 

Extremely 

acceptable 

acceptable 

Borderline 

unacceptable 

Unacceptable 

1 

2 

3 

4 

5 
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POST-TEST  QUESTIONNAIRE 

SUBJECT  NUMBER: _ 

DATE: _ 

EXPERIMENTER:  JJ  CD  LQ 

All  information  will  remain  confidential. 


1.  Rank  order  your  preference  for  a  display  when  referencing  Job  Guides  on  the  flightline 
(1  is  first  choice,  3  is  last  choice): 

Choices  are:  a)  Panasonic  Toughbook  34,  b)  LXE-  MX3  handheld  computer,  c)  Intermec  handheld 
computer,  710  series 

1.  _ 

2.  _ 

3.  _ 

Comments: 


2.  Rank  order  your  preference  for  manipulating  data  (scrolling,  paging,  zooming,  etc.)  when  referencing 
Job  Guides  on  the  flightline  (1  is  first  choice,  3  is  last  choice): 

Choices  are:  a)  Panasonic  Toughbook  34,  b)  LXE-  MX3  handheld  computer,  c)  Intermec  handheld 
computer,  710  series 

1.  _ 

2.  _ 

3.  _ 

Comments: 
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3.  Rank  order  your  preference  for  size  of  the  device  when  referencing  Job  Guides  on  the  flightline  (1  is 
first  choice,  3  is  last  choice): 

Choices  are:  a)  Panasonic  Toughbook  34,  b)  LXE-  MX3  handheld  computer,  c)  Intermec  handheld 
computer,  710  series 

1.  _ 

2.  _ 

3.  _ 

Comments: 


4.  Overall,  rank  order  your  preference  for  one  overall  technology  for  referencing  Job  Guides  on  the 
flightline  (1  is  first  choice,  3  is  last  choice): 

Choices  are:  a)  Panasonic  Toughbook  34,  b)  LXE-  MX3  handheld  computer,  c)  Intermec  handheld 
computer,  710  series 

1.  _ 

2.  _ 

3.  _ 

Comments: 


4.  What  other  comments  do  you  have? 
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